2023-12-21T03:23:43.071 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-21T03:23:43.072 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050 --verbose --name yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050/orig.config.yaml 2023-12-21T03:23:43.142 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-21T03:23:43.257 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050 branch: wip-yuri7-testing-2023-12-20-0808-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7499050' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri7-testing-2023-12-20-0808-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 639f0137318def3e5f393094e5494880043a8f32 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 639f0137318def3e5f393094e5494880043a8f32 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri7-testing-2023-12-20-0808-pacific sha1: 639f0137318def3e5f393094e5494880043a8f32 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 592 sha1: 639f0137318def3e5f393094e5494880043a8f32 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri7-testing-2023-12-20-0808-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 639f0137318def3e5f393094e5494880043a8f32 targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLB67fRCx7IGXvwXku5ZxZXZhxC4PAwTiz0aLgXJ+n6xIJKA/R+CC+J9OQzwkIFbkdiLq5yNDGyUz2FEmUbr2EI= smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJDqnnwX0RmahZB89gv7kXzLnOfDTZrhPFy++7LcxJyHtGk7gepq0QT8PDnO8L5uJ41QKnES5EEOvU0oDScyl90= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-20_19:57:16 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-12-21T03:23:43.257 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa; will attempt to use it 2023-12-21T03:23:43.258 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa/tasks 2023-12-21T03:23:43.258 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-21T03:23:43.321 INFO:teuthology.task.internal:Checking packages... 2023-12-21T03:23:43.347 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '639f0137318def3e5f393094e5494880043a8f32' 2023-12-21T03:23:43.348 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-21T03:23:43.348 INFO:teuthology.packaging:ref: None 2023-12-21T03:23:43.348 INFO:teuthology.packaging:tag: None 2023-12-21T03:23:43.348 INFO:teuthology.packaging:branch: wip-yuri7-testing-2023-12-20-0808-pacific 2023-12-21T03:23:43.348 INFO:teuthology.packaging:sha1: 639f0137318def3e5f393094e5494880043a8f32 2023-12-21T03:23:43.348 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri7-testing-2023-12-20-0808-pacific 2023-12-21T03:23:43.578 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-413.g639f0137 2023-12-21T03:23:43.579 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-21T03:23:43.622 INFO:teuthology.task.internal:no buildpackages task found 2023-12-21T03:23:43.622 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-21T03:23:43.643 INFO:teuthology.task.internal:Saving configuration 2023-12-21T03:23:43.665 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-21T03:23:43.670 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-21T03:23:43.696 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-21 03:18:26.736310', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLB67fRCx7IGXvwXku5ZxZXZhxC4PAwTiz0aLgXJ+n6xIJKA/R+CC+J9OQzwkIFbkdiLq5yNDGyUz2FEmUbr2EI='} 2023-12-21T03:23:43.719 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-21 03:18:26.737734', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJDqnnwX0RmahZB89gv7kXzLnOfDTZrhPFy++7LcxJyHtGk7gepq0QT8PDnO8L5uJ41QKnES5EEOvU0oDScyl90='} 2023-12-21T03:23:43.719 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-21T03:23:43.725 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-12-21T03:23:43.725 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-12-21T03:23:43.725 INFO:teuthology.run_tasks:Running task console_log... 2023-12-21T03:23:43.940 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fad0c9e1e50>, signals=[15]) 2023-12-21T03:23:43.941 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-21T03:23:43.948 INFO:teuthology.task.internal:Opening connections... 2023-12-21T03:23:43.949 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2023-12-21T03:23:43.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-21T03:23:44.020 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2023-12-21T03:23:44.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-21T03:23:44.087 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-21T03:23:44.093 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2023-12-21T03:23:44.116 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2023-12-21T03:23:44.117 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2023-12-21T03:23:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2023-12-21T03:23:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8" 2023-12-21T03:23:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2023-12-21T03:23:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2023-12-21T03:23:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8" 2023-12-21T03:23:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8" 2023-12-21T03:23:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-21T03:23:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2023-12-21T03:23:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-21T03:23:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2023-12-21T03:23:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-21T03:23:44.178 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-21T03:23:44.178 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-21T03:23:44.178 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2023-12-21T03:23:44.203 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2023-12-21T03:23:44.228 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2023-12-21T03:23:44.229 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2023-12-21T03:23:44.289 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-21T03:23:44.290 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-21T03:23:44.291 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2023-12-21T03:23:44.311 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-21T03:23:44.320 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-21T03:23:44.377 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-21T03:23:44.377 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2023-12-21T03:23:44.379 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2023-12-21T03:23:44.400 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-21T03:23:44.406 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-21T03:23:44.406 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2023-12-21T03:23:44.442 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2023-12-21T03:23:44.475 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-21T03:23:44.525 INFO:teuthology.run_tasks:Running task kernel... 2023-12-21T03:23:44.542 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-21T03:23:44.542 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-21T03:23:44.542 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-21T03:23:44.543 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-21T03:23:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-21T03:23:44.563 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2023-12-21T03:23:44.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-21T03:23:44.564 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2023-12-21T03:23:44.625 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-489.el8.x86_64 2023-12-21T03:23:44.625 INFO:teuthology.task.kernel:Running kernel on smithi148: 4.18.0-489.el8.x86_64 2023-12-21T03:23:44.625 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2023-12-21T03:23:44.627 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-489.el8.x86_64 2023-12-21T03:23:44.627 INFO:teuthology.task.kernel:Running kernel on smithi078: 4.18.0-489.el8.x86_64 2023-12-21T03:23:44.627 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2023-12-21T03:23:50.577 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - AppStream 10 MB/s | 35 MB 00:03 2023-12-21T03:23:51.100 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - AppStream 8.9 MB/s | 35 MB 00:03 2023-12-21T03:23:58.928 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - BaseOS 30 MB/s | 57 MB 00:01 2023-12-21T03:24:00.241 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - BaseOS 21 MB/s | 57 MB 00:02 2023-12-21T03:24:07.402 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras 51 kB/s | 18 kB 00:00 2023-12-21T03:24:08.236 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras common packages 11 kB/s | 6.9 kB 00:00 2023-12-21T03:24:08.521 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-12-21T03:24:08.897 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2023-12-21T03:24:09.658 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2023-12-21T03:24:09.951 INFO:teuthology.orchestra.run.smithi148.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-12-21T03:24:10.105 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-12-21T03:24:11.321 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-12-21T03:24:14.064 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 403 kB/s | 24 kB 00:00 2023-12-21T03:24:15.235 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 438 kB/s | 24 kB 00:00 2023-12-21T03:24:17.788 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-21T03:24:17.788 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-21T03:24:18.081 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:24:18.082 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:24:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-12-21T03:24:18.082 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:24:18.082 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:24:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 4.18.0-529.el8 baseos 10 M 2023-12-21T03:24:18.083 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-12-21T03:24:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 4.18.0-529.el8 baseos 43 M 2023-12-21T03:24:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 4.18.0-529.el8 baseos 36 M 2023-12-21T03:24:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:24:18.083 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:24:18.083 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:24:18.083 INFO:teuthology.orchestra.run.smithi078.stdout:Install 3 Packages 2023-12-21T03:24:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:24:18.084 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 89 M 2023-12-21T03:24:18.084 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 96 M 2023-12-21T03:24:18.085 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:24:19.010 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-21T03:24:19.010 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-21T03:24:19.310 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:24:19.311 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:24:19.312 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-12-21T03:24:19.312 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:24:19.312 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:24:19.312 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 4.18.0-529.el8 baseos 10 M 2023-12-21T03:24:19.312 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-12-21T03:24:19.312 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core x86_64 4.18.0-529.el8 baseos 43 M 2023-12-21T03:24:19.312 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules x86_64 4.18.0-529.el8 baseos 36 M 2023-12-21T03:24:19.313 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:24:19.313 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:24:19.313 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:24:19.313 INFO:teuthology.orchestra.run.smithi148.stdout:Install 3 Packages 2023-12-21T03:24:19.313 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:24:19.313 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 89 M 2023-12-21T03:24:19.313 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 96 M 2023-12-21T03:24:19.314 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:24:21.620 INFO:teuthology.orchestra.run.smithi148.stdout:(1/3): kernel-4.18.0-529.el8.x86_64.rpm 5.7 MB/s | 10 MB 00:01 2023-12-21T03:24:21.831 INFO:teuthology.orchestra.run.smithi078.stdout:(1/3): kernel-4.18.0-529.el8.x86_64.rpm 5.6 MB/s | 10 MB 00:01 2023-12-21T03:24:23.207 INFO:teuthology.orchestra.run.smithi078.stdout:(2/3): kernel-modules-4.18.0-529.el8.x86_64.rpm 11 MB/s | 36 MB 00:03 2023-12-21T03:24:24.853 INFO:teuthology.orchestra.run.smithi148.stdout:(2/3): kernel-modules-4.18.0-529.el8.x86_64.rpm 7.2 MB/s | 36 MB 00:05 2023-12-21T03:24:25.612 INFO:teuthology.orchestra.run.smithi148.stdout:(3/3): kernel-core-4.18.0-529.el8.x86_64.rpm 7.5 MB/s | 43 MB 00:05 2023-12-21T03:24:25.612 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:24:25.612 INFO:teuthology.orchestra.run.smithi148.stdout:Total 14 MB/s | 89 MB 00:06 2023-12-21T03:24:26.292 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:24:26.710 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:24:26.711 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:24:27.958 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:24:27.987 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:24:30.816 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:24:31.417 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-21T03:24:32.223 INFO:teuthology.orchestra.run.smithi078.stdout:(3/3): kernel-core-4.18.0-529.el8.x86_64.rpm 3.5 MB/s | 43 MB 00:12 2023-12-21T03:24:32.223 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:24:32.223 INFO:teuthology.orchestra.run.smithi078.stdout:Total 6.3 MB/s | 89 MB 00:14 2023-12-21T03:24:32.905 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:24:33.310 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:24:33.311 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:24:34.546 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:24:34.570 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:24:35.076 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-21T03:24:35.441 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-21T03:24:37.379 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:24:38.027 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-21T03:24:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-21T03:24:39.171 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-4.18.0-529.el8.x86_64 3/3 2023-12-21T03:24:41.590 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-21T03:24:42.043 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-21T03:24:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-21T03:24:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-4.18.0-529.el8.x86_64 3/3 2023-12-21T03:25:01.141 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 3/3 2023-12-21T03:25:01.167 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-21T03:25:01.546 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-4.18.0-529.el8.x86_64 3/3 2023-12-21T03:25:01.546 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-4.18.0-529.el8.x86_64 1/3 2023-12-21T03:25:01.546 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-core-4.18.0-529.el8.x86_64 2/3 2023-12-21T03:25:02.415 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-21T03:25:02.415 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:25:02.415 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:25:02.416 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-4.18.0-529.el8.x86_64 kernel-core-4.18.0-529.el8.x86_64 2023-12-21T03:25:02.416 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-4.18.0-529.el8.x86_64 2023-12-21T03:25:02.416 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:25:02.416 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:25:03.186 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2023-12-21T03:25:03.406 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-21T03:25:03.407 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-21T03:25:03.407 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-529.el8.x86_64 2023-12-21T03:25:03.407 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-21T03:25:03.407 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-21T03:25:03.407 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-12-21T03:25:03.407 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2023-12-21T03:25:03.408 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2023-12-21T03:25:03.618 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-21T03:25:03.618 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-21T03:25:03.618 DEBUG:teuthology.orchestra.run.smithi148:> sudo rpm -qi grub2-tools 2023-12-21T03:25:03.652 INFO:teuthology.orchestra.run.smithi148.stdout:Name : grub2-tools 2023-12-21T03:25:03.652 INFO:teuthology.orchestra.run.smithi148.stdout:Epoch : 1 2023-12-21T03:25:03.652 INFO:teuthology.orchestra.run.smithi148.stdout:Version : 2.02 2023-12-21T03:25:03.652 INFO:teuthology.orchestra.run.smithi148.stdout:Release : 129.el8 2023-12-21T03:25:03.652 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: x86_64 2023-12-21T03:25:03.652 INFO:teuthology.orchestra.run.smithi148.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-21T03:25:03.652 INFO:teuthology.orchestra.run.smithi148.stdout:Group : System Environment/Base 2023-12-21T03:25:03.652 INFO:teuthology.orchestra.run.smithi148.stdout:Size : 9265586 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:License : GPLv3+ 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:Relocations : (not relocatable) 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:Packager : CentOS BuildSystem 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:Vendor : CentOS 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:Summary : Support tools for GRUB. 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:Description : 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-21T03:25:03.653 INFO:teuthology.orchestra.run.smithi148.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-21T03:25:03.654 INFO:teuthology.orchestra.run.smithi148.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-21T03:25:03.654 INFO:teuthology.orchestra.run.smithi148.stdout:hardware devices. 2023-12-21T03:25:03.654 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:25:03.654 INFO:teuthology.orchestra.run.smithi148.stdout:This subpackage provides tools for support of all platforms. 2023-12-21T03:25:03.655 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-21T03:25:03.655 INFO:teuthology.task.kernel:Updating grub on smithi148 to boot 4.18.0-529.el8.x86_64 2023-12-21T03:25:03.655 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-21T03:25:05.014 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2023-12-21T03:25:07.450 INFO:teuthology.orchestra.run.smithi148.stderr:done 2023-12-21T03:25:07.452 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2023-12-21T03:25:07.486 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.zXRnm8j9ys 2023-12-21T03:25:07.486 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zXRnm8j9ys 2023-12-21T03:25:07.541 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 3/3 2023-12-21T03:25:07.560 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.zXRnm8j9ys 2023-12-21T03:25:07.564 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-21T03:25:07.714 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.zXRnm8j9ys is 6KB 2023-12-21T03:25:07.764 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.zXRnm8j9ys 2023-12-21T03:25:07.783 DEBUG:teuthology.orchestra.run.smithi148:> sudo /bin/ls /boot/loader/entries || true 2023-12-21T03:25:07.847 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-21T03:25:07.847 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-21T03:25:07.848 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-21T03:25:07.848 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64.conf 2023-12-21T03:25:07.849 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64 2023-12-21T03:25:07.892 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-4.18.0-529.el8.x86_64 3/3 2023-12-21T03:25:07.892 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-4.18.0-529.el8.x86_64 1/3 2023-12-21T03:25:07.892 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-4.18.0-529.el8.x86_64 2/3 2023-12-21T03:25:08.022 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2023-12-21T03:25:08.663 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-21T03:25:08.663 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:25:08.664 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:25:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-4.18.0-529.el8.x86_64 kernel-core-4.18.0-529.el8.x86_64 2023-12-21T03:25:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-4.18.0-529.el8.x86_64 2023-12-21T03:25:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:25:08.664 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:25:08.868 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2023-12-21T03:25:09.086 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-21T03:25:09.086 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-21T03:25:09.086 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-529.el8.x86_64 2023-12-21T03:25:09.086 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-21T03:25:09.086 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-21T03:25:09.086 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-21T03:25:09.086 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2023-12-21T03:25:09.086 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2023-12-21T03:25:09.296 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-21T03:25:09.296 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-21T03:25:09.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2023-12-21T03:25:09.330 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2023-12-21T03:25:09.330 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2023-12-21T03:25:09.330 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.02 2023-12-21T03:25:09.330 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 129.el8 2023-12-21T03:25:09.330 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2023-12-21T03:25:09.330 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-21T03:25:09.330 INFO:teuthology.orchestra.run.smithi078.stdout:Group : System Environment/Base 2023-12-21T03:25:09.330 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 9265586 2023-12-21T03:25:09.330 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:Relocations : (not relocatable) 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : CentOS BuildSystem 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-21T03:25:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2023-12-21T03:25:09.332 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:25:09.332 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2023-12-21T03:25:09.332 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-21T03:25:09.333 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 4.18.0-529.el8.x86_64 2023-12-21T03:25:09.333 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-21T03:25:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2023-12-21T03:25:12.943 INFO:teuthology.orchestra.run.smithi078.stderr:done 2023-12-21T03:25:12.945 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2023-12-21T03:25:12.968 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.JF5qOgPYHt 2023-12-21T03:25:12.968 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JF5qOgPYHt 2023-12-21T03:25:13.043 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.JF5qOgPYHt 2023-12-21T03:25:13.199 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.JF5qOgPYHt is 6KB 2023-12-21T03:25:13.247 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.JF5qOgPYHt 2023-12-21T03:25:13.265 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2023-12-21T03:25:13.335 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-21T03:25:13.335 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-21T03:25:13.335 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-21T03:25:13.335 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64.conf 2023-12-21T03:25:13.336 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64 2023-12-21T03:25:13.464 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2023-12-21T03:25:38.045 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-12-21T03:25:38.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-21T03:25:38.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-21T03:25:43.471 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-21T03:25:43.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-12-21T03:25:43.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-21T03:26:12.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-12-21T03:26:18.129 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2023-12-21T03:26:21.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-21T03:26:21.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-21T03:26:27.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-12-21T03:26:27.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-21T03:26:39.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-12-21T03:26:45.521 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2023-12-21T03:26:51.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-21T03:26:51.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-21T03:26:53.654 DEBUG:teuthology.orchestra.run.smithi148:> true 2023-12-21T03:26:53.988 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-21T03:26:53.989 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-529.el8.x86_64"... 2023-12-21T03:26:53.989 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2023-12-21T03:26:54.009 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-529.el8.x86_64 2023-12-21T03:26:54.010 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-529.el8.x86_64 vs 4.18.0-529.el8.x86_64 2023-12-21T03:26:54.010 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-21T03:26:54.010 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-21T03:26:55.012 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-12-21T03:26:55.012 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-21T03:26:55.096 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2023-12-21T03:26:55.108 DEBUG:teuthology.parallel:result is None 2023-12-21T03:26:57.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-12-21T03:26:57.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-21T03:26:57.796 DEBUG:teuthology.orchestra.run.smithi078:> true 2023-12-21T03:26:58.082 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-12-21T03:26:58.082 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-529.el8.x86_64"... 2023-12-21T03:26:58.082 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2023-12-21T03:26:58.099 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-529.el8.x86_64 2023-12-21T03:26:58.099 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-529.el8.x86_64 vs 4.18.0-529.el8.x86_64 2023-12-21T03:26:58.099 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-21T03:26:58.099 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-21T03:26:59.100 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-21T03:26:59.101 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-21T03:26:59.198 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2023-12-21T03:26:59.210 DEBUG:teuthology.parallel:result is None 2023-12-21T03:26:59.210 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-21T03:26:59.217 INFO:teuthology.task.internal:Creating test directory... 2023-12-21T03:26:59.217 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-21T03:26:59.219 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-21T03:26:59.239 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-21T03:26:59.252 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-21T03:26:59.285 INFO:teuthology.task.internal:Creating archive directory... 2023-12-21T03:26:59.285 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-21T03:26:59.289 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-21T03:26:59.313 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-21T03:26:59.319 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-21T03:26:59.320 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-21T03:26:59.343 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-21T03:26:59.387 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-21T03:26:59.397 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-21T03:26:59.416 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-21T03:26:59.427 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-21T03:26:59.428 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-21T03:26:59.438 INFO:teuthology.task.internal:Configuring sudo... 2023-12-21T03:26:59.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-21T03:26:59.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-21T03:26:59.469 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-21T03:26:59.570 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-21T03:26:59.570 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-21T03:26:59.573 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-21T03:26:59.590 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-21T03:26:59.673 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-21T03:26:59.713 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-21T03:26:59.795 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-21T03:26:59.833 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:26:59.834 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-21T03:26:59.897 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-21T03:26:59.941 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-21T03:26:59.997 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-21T03:27:00.083 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-21T03:27:00.128 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:27:00.128 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-21T03:27:00.197 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2023-12-21T03:27:00.199 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2023-12-21T03:27:00.266 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-21T03:27:00.350 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-21T03:27:00.654 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-21T03:27:00.662 INFO:teuthology.task.internal:Starting timer... 2023-12-21T03:27:00.662 INFO:teuthology.run_tasks:Running task pcp... 2023-12-21T03:27:00.698 INFO:teuthology.run_tasks:Running task selinux... 2023-12-21T03:27:00.735 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-21T03:27:00.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2023-12-21T03:27:00.886 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2023-12-21T03:27:00.888 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2023-12-21T03:27:01.008 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: 2023-12-21T03:27:01.012 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-21T03:27:01.012 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2023-12-21T03:27:01.102 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2023-12-21T03:27:01.102 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2023-12-21T03:27:01.166 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2023-12-21T03:27:01.166 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi148.front.sepia.ceph.com': 'permissive'} 2023-12-21T03:27:01.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-21T03:27:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-21T03:27:01.211 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-21T03:27:01.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-21T03:27:01.479 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-21T03:27:01.479 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2023-12-21T03:27:01.845 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2023-12-21T03:27:02.488 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-21T03:27:02.498 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-21T03:27:02.498 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-21T03:27:02.516 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-21T03:27:02.517 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi148.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-21T03:34:30.430 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi148.front.sepia.ceph.com')] 2023-12-21T03:34:30.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-12-21T03:34:30.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-21T03:34:30.504 DEBUG:teuthology.orchestra.run.smithi078:> true 2023-12-21T03:34:30.572 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-12-21T03:34:30.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-21T03:34:30.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-21T03:34:30.642 DEBUG:teuthology.orchestra.run.smithi148:> true 2023-12-21T03:34:30.712 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-21T03:34:30.713 INFO:teuthology.run_tasks:Running task clock... 2023-12-21T03:34:30.729 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-21T03:34:30.730 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-21T03:34:30.730 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-21T03:34:30.733 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-21T03:34:30.733 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-21T03:34:30.764 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-21T03:34:30.778 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-21T03:34:30.786 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-21T03:34:30.800 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-21T03:34:30.847 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2023-12-21T03:34:30.859 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2023-12-21T03:34:30.865 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: ntpd: command not found 2023-12-21T03:34:30.872 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-21T03:34:30.876 INFO:teuthology.orchestra.run.smithi148.stdout:506 Cannot talk to daemon 2023-12-21T03:34:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-21T03:34:30.891 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-21T03:34:30.905 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-21T03:34:30.953 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2023-12-21T03:34:30.956 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-21T03:34:30.956 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2023-12-21T03:34:30.956 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-21T03:34:30.957 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-21T03:34:30.957 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-21T03:34:30.957 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-21T03:34:30.977 INFO:teuthology.orchestra.run.smithi148.stderr:bash: ntpq: command not found 2023-12-21T03:34:30.980 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-21T03:34:30.980 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2023-12-21T03:34:30.980 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-21T03:34:30.980 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-21T03:34:30.980 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-21T03:34:30.980 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-21T03:34:30.981 INFO:teuthology.run_tasks:Running task pexec... 2023-12-21T03:34:30.992 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-21T03:34:30.992 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi078.front.sepia.ceph.com 2023-12-21T03:34:30.992 DEBUG:teuthology.orchestra.run.smithi078:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-21T03:34:30.993 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi148.front.sepia.ceph.com 2023-12-21T03:34:30.993 DEBUG:teuthology.orchestra.run.smithi148:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-21T03:34:31.871 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:37 ago on Thu 21 Dec 2023 03:32:54 AM UTC. 2023-12-21T03:34:31.898 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:36 ago on Thu 21 Dec 2023 03:32:55 AM UTC. 2023-12-21T03:34:34.013 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:34:34.014 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:34:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-12-21T03:34:34.014 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:34:34.014 INFO:teuthology.orchestra.run.smithi078.stdout:Resetting modules: 2023-12-21T03:34:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools 2023-12-21T03:34:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:34:34.014 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:34:34.014 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:34:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:34:34.091 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:34:34.092 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:34:34.092 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-12-21T03:34:34.092 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:34:34.092 INFO:teuthology.orchestra.run.smithi148.stdout:Resetting modules: 2023-12-21T03:34:34.092 INFO:teuthology.orchestra.run.smithi148.stdout: container-tools 2023-12-21T03:34:34.092 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:34:34.092 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:34:34.092 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:34:34.092 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:34:34.273 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:34:34.357 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:34:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:41 ago on Thu 21 Dec 2023 03:32:54 AM UTC. 2023-12-21T03:34:35.324 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:40 ago on Thu 21 Dec 2023 03:32:55 AM UTC. 2023-12-21T03:34:37.338 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:34:37.469 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:34:37.525 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:34:37.525 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-12-21T03:34:37.525 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:34:37.525 INFO:teuthology.orchestra.run.smithi078.stdout:Installing group/module packages: 2023-12-21T03:34:37.525 INFO:teuthology.orchestra.run.smithi078.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout:Installing weak dependencies: 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-21T03:34:37.526 INFO:teuthology.orchestra.run.smithi078.stdout:Installing module profiles: 2023-12-21T03:34:37.527 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools/common 2023-12-21T03:34:37.527 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:34:37.527 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling module streams: 2023-12-21T03:34:37.527 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools rhel8 2023-12-21T03:34:37.527 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:34:37.527 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:34:37.527 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:34:37.527 INFO:teuthology.orchestra.run.smithi078.stdout:Install 13 Packages 2023-12-21T03:34:37.527 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:34:37.528 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 27 M 2023-12-21T03:34:37.528 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 89 M 2023-12-21T03:34:37.528 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:34:37.658 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:34:37.658 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-12-21T03:34:37.658 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout:Installing group/module packages: 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout:Installing weak dependencies: 2023-12-21T03:34:37.659 INFO:teuthology.orchestra.run.smithi148.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-21T03:34:37.660 INFO:teuthology.orchestra.run.smithi148.stdout:Installing module profiles: 2023-12-21T03:34:37.660 INFO:teuthology.orchestra.run.smithi148.stdout: container-tools/common 2023-12-21T03:34:37.660 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:34:37.660 INFO:teuthology.orchestra.run.smithi148.stdout:Enabling module streams: 2023-12-21T03:34:37.660 INFO:teuthology.orchestra.run.smithi148.stdout: container-tools rhel8 2023-12-21T03:34:37.660 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:34:37.660 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:34:37.660 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:34:37.660 INFO:teuthology.orchestra.run.smithi148.stdout:Install 13 Packages 2023-12-21T03:34:37.660 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:34:37.661 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 27 M 2023-12-21T03:34:37.661 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 89 M 2023-12-21T03:34:37.661 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:34:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.3 MB/s | 750 kB 00:00 2023-12-21T03:34:38.369 INFO:teuthology.orchestra.run.smithi148.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.1 MB/s | 750 kB 00:00 2023-12-21T03:34:38.407 INFO:teuthology.orchestra.run.smithi078.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 448 kB/s | 38 kB 00:00 2023-12-21T03:34:38.428 INFO:teuthology.orchestra.run.smithi148.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 650 kB/s | 38 kB 00:00 2023-12-21T03:34:38.448 INFO:teuthology.orchestra.run.smithi078.stdout:(3/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.4 MB/s | 1.0 MB 00:00 2023-12-21T03:34:38.528 INFO:teuthology.orchestra.run.smithi148.stdout:(3/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.2 MB/s | 1.0 MB 00:00 2023-12-21T03:34:38.549 INFO:teuthology.orchestra.run.smithi078.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.7 MB/s | 241 kB 00:00 2023-12-21T03:34:38.612 INFO:teuthology.orchestra.run.smithi148.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.3 MB/s | 241 kB 00:00 2023-12-21T03:34:38.657 INFO:teuthology.orchestra.run.smithi078.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.6 MB/s | 178 kB 00:00 2023-12-21T03:34:38.716 INFO:teuthology.orchestra.run.smithi078.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 424 kB/s | 25 kB 00:00 2023-12-21T03:34:38.720 INFO:teuthology.orchestra.run.smithi148.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.6 MB/s | 178 kB 00:00 2023-12-21T03:34:38.783 INFO:teuthology.orchestra.run.smithi078.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2023-12-21T03:34:38.788 INFO:teuthology.orchestra.run.smithi148.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 371 kB/s | 25 kB 00:00 2023-12-21T03:34:38.863 INFO:teuthology.orchestra.run.smithi148.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2023-12-21T03:34:39.425 INFO:teuthology.orchestra.run.smithi078.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 4.0 MB/s | 3.9 MB 00:00 2023-12-21T03:34:39.564 INFO:teuthology.orchestra.run.smithi148.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 3.8 MB/s | 3.9 MB 00:01 2023-12-21T03:34:39.759 INFO:teuthology.orchestra.run.smithi078.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 7.6 MB/s | 2.5 MB 00:00 2023-12-21T03:34:39.899 INFO:teuthology.orchestra.run.smithi078.stdout:(10/13): buildah-1.32.0-1.module_el8+712+4cd1bd 4.5 MB/s | 9.6 MB 00:02 2023-12-21T03:34:39.926 INFO:teuthology.orchestra.run.smithi078.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 315 kB/s | 53 kB 00:00 2023-12-21T03:34:39.939 INFO:teuthology.orchestra.run.smithi148.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 6.8 MB/s | 2.5 MB 00:00 2023-12-21T03:34:39.967 INFO:teuthology.orchestra.run.smithi078.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 604 kB/s | 41 kB 00:00 2023-12-21T03:34:40.006 INFO:teuthology.orchestra.run.smithi148.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 791 kB/s | 53 kB 00:00 2023-12-21T03:34:40.103 INFO:teuthology.orchestra.run.smithi148.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 422 kB/s | 41 kB 00:00 2023-12-21T03:34:40.206 INFO:teuthology.orchestra.run.smithi148.stdout:(12/13): buildah-1.32.0-1.module_el8+712+4cd1bd 3.9 MB/s | 9.6 MB 00:02 2023-12-21T03:34:40.260 INFO:teuthology.orchestra.run.smithi078.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 5.5 MB/s | 8.1 MB 00:01 2023-12-21T03:34:40.260 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:34:40.260 INFO:teuthology.orchestra.run.smithi078.stdout:Total 9.7 MB/s | 27 MB 00:02 2023-12-21T03:34:40.357 INFO:teuthology.orchestra.run.smithi148.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 5.5 MB/s | 8.1 MB 00:01 2023-12-21T03:34:40.357 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:34:40.357 INFO:teuthology.orchestra.run.smithi148.stdout:Total 9.9 MB/s | 27 MB 00:02 2023-12-21T03:34:40.530 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:34:40.551 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:34:40.551 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:34:40.625 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:34:40.646 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:34:40.646 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:34:40.808 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:34:40.811 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:34:40.903 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:34:40.906 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:34:41.581 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:34:41.608 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:34:41.802 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-21T03:34:41.850 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-21T03:34:41.972 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-21T03:34:42.112 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-21T03:34:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-21T03:34:42.297 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-21T03:34:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-21T03:34:42.539 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-21T03:34:42.931 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-21T03:34:43.104 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-21T03:34:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-21T03:34:44.111 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-21T03:34:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-21T03:34:44.333 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-21T03:34:44.355 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-21T03:34:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-21T03:34:44.597 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-21T03:34:44.882 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-21T03:34:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-21T03:34:45.274 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-21T03:34:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-21T03:34:46.132 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-21T03:34:46.188 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-21T03:34:46.450 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-21T03:34:46.635 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-21T03:34:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-21T03:34:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-21T03:34:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-21T03:34:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-21T03:34:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-21T03:34:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-21T03:34:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-21T03:34:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-21T03:34:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-21T03:34:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-21T03:34:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-21T03:34:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-21T03:34:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-21T03:34:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-21T03:34:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-21T03:34:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-21T03:34:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-21T03:34:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-21T03:34:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-21T03:34:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-21T03:34:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-21T03:34:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-21T03:34:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-21T03:34:47.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-21T03:34:47.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-21T03:34:47.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-21T03:34:47.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-21T03:34:48.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-21T03:34:48.088 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:34:48.089 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:34:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-21T03:34:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-21T03:34:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-21T03:34:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-21T03:34:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-21T03:34:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-21T03:34:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-21T03:34:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:34:48.090 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:34:48.309 DEBUG:teuthology.parallel:result is None 2023-12-21T03:34:48.514 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-21T03:34:48.514 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:34:48.514 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:34:48.514 INFO:teuthology.orchestra.run.smithi148.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.514 INFO:teuthology.orchestra.run.smithi148.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.514 INFO:teuthology.orchestra.run.smithi148.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-21T03:34:48.514 INFO:teuthology.orchestra.run.smithi148.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-21T03:34:48.515 INFO:teuthology.orchestra.run.smithi148.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.515 INFO:teuthology.orchestra.run.smithi148.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.515 INFO:teuthology.orchestra.run.smithi148.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-21T03:34:48.515 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-21T03:34:48.515 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-21T03:34:48.515 INFO:teuthology.orchestra.run.smithi148.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.515 INFO:teuthology.orchestra.run.smithi148.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-21T03:34:48.515 INFO:teuthology.orchestra.run.smithi148.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-21T03:34:48.515 INFO:teuthology.orchestra.run.smithi148.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-21T03:34:48.515 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:34:48.515 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:34:48.813 DEBUG:teuthology.parallel:result is None 2023-12-21T03:34:48.813 INFO:teuthology.run_tasks:Running task pexec... 2023-12-21T03:34:48.822 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-21T03:34:48.825 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi078.front.sepia.ceph.com 2023-12-21T03:34:48.825 DEBUG:teuthology.orchestra.run.smithi078:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-21T03:34:48.825 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi148.front.sepia.ceph.com 2023-12-21T03:34:48.826 DEBUG:teuthology.orchestra.run.smithi148:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-21T03:34:48.975 DEBUG:teuthology.parallel:result is None 2023-12-21T03:34:49.001 DEBUG:teuthology.parallel:result is None 2023-12-21T03:34:49.001 INFO:teuthology.run_tasks:Running task install... 2023-12-21T03:34:49.011 DEBUG:teuthology.task.install:project ceph 2023-12-21T03:34:49.011 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '639f0137318def3e5f393094e5494880043a8f32'}} 2023-12-21T03:34:49.011 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '639f0137318def3e5f393094e5494880043a8f32'} 2023-12-21T03:34:49.011 INFO:teuthology.task.install:Using flavor: default 2023-12-21T03:34:49.019 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-21T03:34:49.019 INFO:teuthology.task.install:extra packages: [] 2023-12-21T03:34:49.019 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '639f0137318def3e5f393094e5494880043a8f32', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-21T03:34:49.019 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-21T03:34:49.020 INFO:teuthology.packaging:ref: None 2023-12-21T03:34:49.020 INFO:teuthology.packaging:tag: v16.2.4 2023-12-21T03:34:49.020 INFO:teuthology.packaging:branch: None 2023-12-21T03:34:49.020 INFO:teuthology.packaging:sha1: 639f0137318def3e5f393094e5494880043a8f32 2023-12-21T03:34:49.214 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-21T03:34:49.215 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-21T03:34:49.218 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '639f0137318def3e5f393094e5494880043a8f32', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-21T03:34:49.219 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-21T03:34:49.220 INFO:teuthology.packaging:ref: None 2023-12-21T03:34:49.220 INFO:teuthology.packaging:tag: v16.2.4 2023-12-21T03:34:49.220 INFO:teuthology.packaging:branch: None 2023-12-21T03:34:49.220 INFO:teuthology.packaging:sha1: 639f0137318def3e5f393094e5494880043a8f32 2023-12-21T03:34:49.429 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-21T03:34:49.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-21T03:34:49.585 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-21T03:34:49.585 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-21T03:34:49.616 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-21T03:34:49.617 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:34:49.617 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-21T03:34:49.646 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-21T03:34:49.646 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-21T03:34:49.646 INFO:teuthology.packaging:ref: None 2023-12-21T03:34:49.646 INFO:teuthology.packaging:tag: v16.2.4 2023-12-21T03:34:49.646 INFO:teuthology.packaging:branch: None 2023-12-21T03:34:49.646 INFO:teuthology.packaging:sha1: 639f0137318def3e5f393094e5494880043a8f32 2023-12-21T03:34:49.646 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-21T03:34:49.670 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-21T03:34:49.670 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-21T03:34:49.695 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-21T03:34:49.696 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:34:49.696 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-21T03:34:49.716 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-21T03:34:49.726 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-21T03:34:49.727 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-21T03:34:49.727 INFO:teuthology.packaging:ref: None 2023-12-21T03:34:49.727 INFO:teuthology.packaging:tag: v16.2.4 2023-12-21T03:34:49.727 INFO:teuthology.packaging:branch: None 2023-12-21T03:34:49.727 INFO:teuthology.packaging:sha1: 639f0137318def3e5f393094e5494880043a8f32 2023-12-21T03:34:49.728 DEBUG:teuthology.orchestra.run.smithi148:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-21T03:34:49.793 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-21T03:34:49.795 DEBUG:teuthology.orchestra.run.smithi148:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-21T03:34:49.860 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2023-12-21T03:34:49.861 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2023-12-21T03:34:49.873 DEBUG:teuthology.orchestra.run.smithi148:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-21T03:34:49.940 INFO:teuthology.orchestra.run.smithi148.stdout:check_obsoletes = 1 2023-12-21T03:34:49.942 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum clean all 2023-12-21T03:34:50.130 INFO:teuthology.orchestra.run.smithi078.stdout:76 files removed 2023-12-21T03:34:50.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw 2023-12-21T03:34:50.216 INFO:teuthology.orchestra.run.smithi148.stdout:76 files removed 2023-12-21T03:34:50.244 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-radosgw 2023-12-21T03:34:50.676 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 671 kB/s | 74 kB 00:00 2023-12-21T03:34:50.740 INFO:teuthology.orchestra.run.smithi148.stdout:ceph packages for x86_64 838 kB/s | 74 kB 00:00 2023-12-21T03:34:50.827 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 134 kB/s | 15 kB 00:00 2023-12-21T03:34:50.872 INFO:teuthology.orchestra.run.smithi148.stdout:ceph noarch packages 162 kB/s | 15 kB 00:00 2023-12-21T03:34:50.961 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 17 kB/s | 1.7 kB 00:00 2023-12-21T03:34:50.992 INFO:teuthology.orchestra.run.smithi148.stdout:ceph source packages 19 kB/s | 1.7 kB 00:00 2023-12-21T03:34:52.937 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - AppStream 18 MB/s | 35 MB 00:01 2023-12-21T03:34:53.270 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - AppStream 15 MB/s | 35 MB 00:02 2023-12-21T03:35:01.763 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 57 MB 00:02 2023-12-21T03:35:01.993 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - BaseOS 24 MB/s | 57 MB 00:02 2023-12-21T03:35:10.371 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras 35 kB/s | 18 kB 00:00 2023-12-21T03:35:11.559 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras 12 kB/s | 18 kB 00:01 2023-12-21T03:35:11.957 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras common packages 4.9 kB/s | 6.9 kB 00:01 2023-12-21T03:35:12.118 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-12-21T03:35:12.581 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - PowerTools 14 MB/s | 6.6 MB 00:00 2023-12-21T03:35:13.127 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - PowerTools 7.8 MB/s | 6.6 MB 00:00 2023-12-21T03:35:14.074 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-12-21T03:35:14.662 INFO:teuthology.orchestra.run.smithi148.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-12-21T03:35:15.897 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2023-12-21T03:35:16.175 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 16 MB 00:01 2023-12-21T03:35:19.831 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 182 kB/s | 24 kB 00:00 2023-12-21T03:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 427 kB/s | 24 kB 00:00 2023-12-21T03:35:24.465 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:35:24.468 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:35:24.468 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-12-21T03:35:24.468 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:35:24.468 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:35:24.468 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-21T03:35:24.468 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrading: 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-21T03:35:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-21T03:35:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:35:24.471 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:35:24.471 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:35:24.471 INFO:teuthology.orchestra.run.smithi148.stdout:Install 25 Packages 2023-12-21T03:35:24.471 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrade 2 Packages 2023-12-21T03:35:24.471 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:35:24.471 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 59 M 2023-12-21T03:35:24.471 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:35:24.732 INFO:teuthology.orchestra.run.smithi148.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 41 MB/s | 6.0 MB 00:00 2023-12-21T03:35:24.749 INFO:teuthology.orchestra.run.smithi148.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2023-12-21T03:35:24.759 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:35:24.762 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:35:24.762 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-12-21T03:35:24.762 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:35:24.762 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:35:24.762 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-21T03:35:24.763 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-21T03:35:24.764 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-21T03:35:24.765 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:35:24.765 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:35:24.765 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:35:24.765 INFO:teuthology.orchestra.run.smithi078.stdout:Install 25 Packages 2023-12-21T03:35:24.765 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 2 Packages 2023-12-21T03:35:24.765 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:35:24.765 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 59 M 2023-12-21T03:35:24.766 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:35:24.782 INFO:teuthology.orchestra.run.smithi148.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2023-12-21T03:35:24.824 INFO:teuthology.orchestra.run.smithi148.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2023-12-21T03:35:24.968 INFO:teuthology.orchestra.run.smithi148.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 29 MB/s | 4.1 MB 00:00 2023-12-21T03:35:25.083 INFO:teuthology.orchestra.run.smithi148.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 383 kB/s | 44 kB 00:00 2023-12-21T03:35:25.108 INFO:teuthology.orchestra.run.smithi148.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-12-21T03:35:25.134 INFO:teuthology.orchestra.run.smithi148.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-12-21T03:35:25.167 INFO:teuthology.orchestra.run.smithi148.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-12-21T03:35:25.209 INFO:teuthology.orchestra.run.smithi148.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 8.6 MB/s | 368 kB 00:00 2023-12-21T03:35:25.235 INFO:teuthology.orchestra.run.smithi148.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-12-21T03:35:25.393 INFO:teuthology.orchestra.run.smithi148.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 16 MB/s | 13 MB 00:00 2023-12-21T03:35:25.658 INFO:teuthology.orchestra.run.smithi078.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 34 MB/s | 6.0 MB 00:00 2023-12-21T03:35:25.677 INFO:teuthology.orchestra.run.smithi148.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 22 MB/s | 24 MB 00:01 2023-12-21T03:35:25.686 INFO:teuthology.orchestra.run.smithi078.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 892 kB/s | 25 kB 00:00 2023-12-21T03:35:25.702 INFO:teuthology.orchestra.run.smithi148.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 311 kB/s | 145 kB 00:00 2023-12-21T03:35:25.719 INFO:teuthology.orchestra.run.smithi078.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2023-12-21T03:35:25.753 INFO:teuthology.orchestra.run.smithi078.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-12-21T03:35:25.769 INFO:teuthology.orchestra.run.smithi148.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 659 kB/s | 44 kB 00:00 2023-12-21T03:35:25.853 INFO:teuthology.orchestra.run.smithi078.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2023-12-21T03:35:25.878 INFO:teuthology.orchestra.run.smithi078.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-12-21T03:35:25.904 INFO:teuthology.orchestra.run.smithi078.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-12-21T03:35:25.919 INFO:teuthology.orchestra.run.smithi148.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 281 kB/s | 42 kB 00:00 2023-12-21T03:35:25.921 INFO:teuthology.orchestra.run.smithi078.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 13 MB/s | 213 kB 00:00 2023-12-21T03:35:25.945 INFO:teuthology.orchestra.run.smithi148.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 970 kB/s | 259 kB 00:00 2023-12-21T03:35:25.946 INFO:teuthology.orchestra.run.smithi078.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 15 MB/s | 387 kB 00:00 2023-12-21T03:35:25.971 INFO:teuthology.orchestra.run.smithi078.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2023-12-21T03:35:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.4 MB/s | 114 kB 00:00 2023-12-21T03:35:26.004 INFO:teuthology.orchestra.run.smithi148.stdout:(18/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.1 MB/s | 677 kB 00:00 2023-12-21T03:35:26.029 INFO:teuthology.orchestra.run.smithi148.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 430 kB/s | 47 kB 00:00 2023-12-21T03:35:26.130 INFO:teuthology.orchestra.run.smithi148.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 436 kB/s | 80 kB 00:00 2023-12-21T03:35:26.164 INFO:teuthology.orchestra.run.smithi078.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2023-12-21T03:35:26.439 INFO:teuthology.orchestra.run.smithi078.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2023-12-21T03:35:26.464 INFO:teuthology.orchestra.run.smithi078.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 311 kB/s | 145 kB 00:00 2023-12-21T03:35:26.480 INFO:teuthology.orchestra.run.smithi148.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 245 kB/s | 116 kB 00:00 2023-12-21T03:35:26.531 INFO:teuthology.orchestra.run.smithi078.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 658 kB/s | 44 kB 00:00 2023-12-21T03:35:26.622 INFO:teuthology.orchestra.run.smithi148.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 368 kB/s | 181 kB 00:00 2023-12-21T03:35:26.647 INFO:teuthology.orchestra.run.smithi148.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 496 kB/s | 306 kB 00:00 2023-12-21T03:35:26.672 INFO:teuthology.orchestra.run.smithi148.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 306 kB/s | 59 kB 00:00 2023-12-21T03:35:26.681 INFO:teuthology.orchestra.run.smithi078.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 281 kB/s | 42 kB 00:00 2023-12-21T03:35:26.707 INFO:teuthology.orchestra.run.smithi078.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 972 kB/s | 259 kB 00:00 2023-12-21T03:35:26.732 INFO:teuthology.orchestra.run.smithi078.stdout:(18/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2023-12-21T03:35:26.757 INFO:teuthology.orchestra.run.smithi078.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 620 kB/s | 47 kB 00:00 2023-12-21T03:35:26.764 INFO:teuthology.orchestra.run.smithi148.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2023-12-21T03:35:26.781 INFO:teuthology.orchestra.run.smithi148.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 472 kB/s | 75 kB 00:00 2023-12-21T03:35:26.839 INFO:teuthology.orchestra.run.smithi148.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2023-12-21T03:35:26.841 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:35:26.842 INFO:teuthology.orchestra.run.smithi148.stdout:Total 25 MB/s | 59 MB 00:02 2023-12-21T03:35:26.874 INFO:teuthology.orchestra.run.smithi078.stdout:(20/27): fmt-6.2.1-1.el8.x86_64.rpm 819 kB/s | 116 kB 00:00 2023-12-21T03:35:26.900 INFO:teuthology.orchestra.run.smithi078.stdout:(21/27): ndctl-libs-71.1-7.el8.x86_64.rpm 416 kB/s | 80 kB 00:00 2023-12-21T03:35:26.911 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:35:26.925 INFO:teuthology.orchestra.run.smithi078.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2023-12-21T03:35:26.950 INFO:teuthology.orchestra.run.smithi078.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 2.3 MB/s | 181 kB 00:00 2023-12-21T03:35:26.961 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:35:26.961 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:35:26.967 INFO:teuthology.orchestra.run.smithi078.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2023-12-21T03:35:27.001 INFO:teuthology.orchestra.run.smithi078.stdout:(25/27): liboath-2.6.2-3.el8.x86_64.rpm 584 kB/s | 59 kB 00:00 2023-12-21T03:35:27.076 INFO:teuthology.orchestra.run.smithi078.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2023-12-21T03:35:27.169 INFO:teuthology.orchestra.run.smithi078.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 20 MB/s | 4.0 MB 00:00 2023-12-21T03:35:27.170 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:35:27.171 INFO:teuthology.orchestra.run.smithi078.stdout:Total 25 MB/s | 59 MB 00:02 2023-12-21T03:35:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:35:27.290 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:35:27.290 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:35:27.310 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:35:27.313 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:35:27.639 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:35:27.641 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:35:28.476 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:35:28.708 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-21T03:35:28.921 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-21T03:35:29.080 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-21T03:35:29.093 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:35:29.298 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-21T03:35:29.503 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-21T03:35:29.529 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-21T03:35:29.705 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-21T03:35:29.708 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-21T03:35:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-21T03:35:30.054 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-21T03:35:30.125 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-21T03:35:30.308 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-21T03:35:30.402 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-21T03:35:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-21T03:35:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-21T03:35:30.620 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-21T03:35:30.716 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-21T03:35:30.775 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-21T03:35:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-21T03:35:31.034 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-21T03:35:31.264 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2023-12-21T03:35:31.429 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-21T03:35:31.509 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-21T03:35:31.583 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2023-12-21T03:35:31.709 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-21T03:35:31.749 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-21T03:35:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-21T03:35:31.863 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-21T03:35:31.924 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-21T03:35:32.004 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-21T03:35:32.126 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-21T03:35:32.199 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-21T03:35:32.262 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-21T03:35:32.330 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-21T03:35:32.430 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-21T03:35:32.512 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-21T03:35:32.607 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-21T03:35:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-21T03:35:32.771 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-21T03:35:32.816 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-21T03:35:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-21T03:35:33.066 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-21T03:35:33.199 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-21T03:35:33.396 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-21T03:35:33.483 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-21T03:35:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-21T03:35:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-21T03:35:33.699 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-21T03:35:33.803 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-21T03:35:33.905 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-21T03:35:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-21T03:35:34.020 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-21T03:35:34.085 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-21T03:35:34.159 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-21T03:35:34.307 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-21T03:35:34.321 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-21T03:35:34.364 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-21T03:35:34.528 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-21T03:35:34.600 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-21T03:35:34.701 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-21T03:35:34.757 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-21T03:35:34.931 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-21T03:35:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-21T03:35:35.158 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-21T03:35:37.486 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-21T03:35:37.703 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-21T03:35:37.754 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-21T03:35:37.924 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-21T03:35:38.449 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-21T03:35:38.629 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-21T03:35:38.763 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-21T03:35:38.861 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-21T03:35:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-21T03:35:39.062 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-21T03:35:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-21T03:35:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-21T03:35:57.326 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-21T03:35:57.326 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /sys 2023-12-21T03:35:57.326 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /proc 2023-12-21T03:35:57.326 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /dev 2023-12-21T03:35:57.326 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /run 2023-12-21T03:35:57.326 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /mnt 2023-12-21T03:35:57.327 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /var/tmp 2023-12-21T03:35:57.327 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /home 2023-12-21T03:35:57.327 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /tmp 2023-12-21T03:35:57.327 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /dev 2023-12-21T03:35:57.327 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:35:57.688 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-21T03:35:57.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2023-12-21T03:35:57.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2023-12-21T03:35:57.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2023-12-21T03:35:57.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /run 2023-12-21T03:35:57.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2023-12-21T03:35:57.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2023-12-21T03:35:57.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2023-12-21T03:35:57.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2023-12-21T03:35:57.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2023-12-21T03:35:57.688 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:35:58.701 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-21T03:35:58.783 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-21T03:35:58.783 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-21T03:35:59.011 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-21T03:35:59.011 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-21T03:35:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-21T03:35:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-21T03:35:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-21T03:35:59.332 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-21T03:35:59.332 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-21T03:35:59.750 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-21T03:35:59.751 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-21T03:35:59.751 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-21T03:35:59.751 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-21T03:35:59.751 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-21T03:35:59.751 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-21T03:35:59.751 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-21T03:35:59.751 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-21T03:35:59.751 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-21T03:35:59.751 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-21T03:35:59.751 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-21T03:35:59.752 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-21T03:35:59.753 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-21T03:35:59.753 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-21T03:35:59.753 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-21T03:35:59.753 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-21T03:35:59.753 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-21T03:35:59.753 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-21T03:36:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-21T03:36:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-21T03:36:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-21T03:36:01.148 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-21T03:36:01.149 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:01.149 INFO:teuthology.orchestra.run.smithi148.stdout:Upgraded: 2023-12-21T03:36:01.149 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.149 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:36:01.149 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.149 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.149 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: leveldb-1.22-1.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka-1.6.1-1.el8.x86_64 2023-12-21T03:36:01.150 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.151 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-21T03:36:01.151 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-21T03:36:01.151 INFO:teuthology.orchestra.run.smithi148.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-21T03:36:01.151 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.151 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.151 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.151 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-21T03:36:01.152 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.152 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.152 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.152 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:01.152 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:36:01.279 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-21T03:36:01.279 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:01.280 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2023-12-21T03:36:01.280 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.280 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:36:01.280 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.280 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.280 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.281 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.281 INFO:teuthology.orchestra.run.smithi078.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-21T03:36:01.281 INFO:teuthology.orchestra.run.smithi078.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-21T03:36:01.281 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-21T03:36:01.281 INFO:teuthology.orchestra.run.smithi078.stdout: leveldb-1.22-1.el8.x86_64 2023-12-21T03:36:01.282 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.282 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-21T03:36:01.282 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-21T03:36:01.282 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-21T03:36:01.282 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.283 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-1.6.1-1.el8.x86_64 2023-12-21T03:36:01.283 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.283 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-21T03:36:01.283 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-21T03:36:01.283 INFO:teuthology.orchestra.run.smithi078.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-21T03:36:01.283 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.283 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.284 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.284 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-21T03:36:01.284 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.284 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.284 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:01.284 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:01.284 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:36:01.320 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-test 2023-12-21T03:36:01.537 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-test 2023-12-21T03:36:02.181 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:43 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:36:02.392 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:42 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:36:03.852 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:03.854 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:36:03.855 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:03.855 INFO:teuthology.orchestra.run.smithi148.stdout:Install 6 Packages 2023-12-21T03:36:03.855 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:03.855 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 53 M 2023-12-21T03:36:03.855 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 229 M 2023-12-21T03:36:03.855 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:36:03.974 INFO:teuthology.orchestra.run.smithi148.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 169 kB 00:00 2023-12-21T03:36:04.035 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:36:04.036 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:04.036 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-12-21T03:36:04.037 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:04.037 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:36:04.037 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-21T03:36:04.037 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-12-21T03:36:04.037 INFO:teuthology.orchestra.run.smithi078.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-21T03:36:04.037 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-21T03:36:04.038 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-21T03:36:04.038 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-21T03:36:04.038 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-21T03:36:04.038 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:04.038 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:36:04.038 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:04.038 INFO:teuthology.orchestra.run.smithi078.stdout:Install 6 Packages 2023-12-21T03:36:04.038 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:04.039 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 53 M 2023-12-21T03:36:04.039 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 229 M 2023-12-21T03:36:04.040 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:36:04.133 INFO:teuthology.orchestra.run.smithi148.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 860 kB/s | 203 kB 00:00 2023-12-21T03:36:04.286 INFO:teuthology.orchestra.run.smithi078.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.0 MB/s | 169 kB 00:00 2023-12-21T03:36:04.470 INFO:teuthology.orchestra.run.smithi078.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 853 kB/s | 203 kB 00:00 2023-12-21T03:36:04.512 INFO:teuthology.orchestra.run.smithi078.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 832 kB/s | 187 kB 00:00 2023-12-21T03:36:04.667 INFO:teuthology.orchestra.run.smithi148.stdout:(3/6): ceph-test-16.2.4-0.el8.x86_64.rpm 67 MB/s | 52 MB 00:00 2023-12-21T03:36:04.692 INFO:teuthology.orchestra.run.smithi148.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 261 kB/s | 187 kB 00:00 2023-12-21T03:36:04.717 INFO:teuthology.orchestra.run.smithi148.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 553 kB/s | 323 kB 00:00 2023-12-21T03:36:04.859 INFO:teuthology.orchestra.run.smithi148.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 369 kB/s | 71 kB 00:00 2023-12-21T03:36:04.860 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:04.860 INFO:teuthology.orchestra.run.smithi148.stdout:Total 53 MB/s | 53 MB 00:01 2023-12-21T03:36:04.904 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:36:04.925 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:36:04.925 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:36:05.038 INFO:teuthology.orchestra.run.smithi078.stdout:(4/6): ceph-test-16.2.4-0.el8.x86_64.rpm 64 MB/s | 52 MB 00:00 2023-12-21T03:36:05.062 INFO:teuthology.orchestra.run.smithi078.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 128 kB/s | 71 kB 00:00 2023-12-21T03:36:05.088 INFO:teuthology.orchestra.run.smithi078.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 523 kB/s | 323 kB 00:00 2023-12-21T03:36:05.090 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:05.090 INFO:teuthology.orchestra.run.smithi078.stdout:Total 50 MB/s | 53 MB 00:01 2023-12-21T03:36:05.133 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:36:05.154 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:36:05.154 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:36:05.235 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:36:05.238 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:36:05.468 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:36:05.471 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:36:05.721 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:36:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:36:05.969 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-21T03:36:06.160 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-21T03:36:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-21T03:36:06.349 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-21T03:36:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-21T03:36:06.430 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-21T03:36:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-21T03:36:06.623 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-21T03:36:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-21T03:36:06.757 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-21T03:36:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-21T03:36:06.995 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-21T03:36:11.669 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-21T03:36:11.857 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-21T03:36:11.924 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-21T03:36:12.084 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-21T03:36:12.202 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-21T03:36:12.202 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-21T03:36:12.202 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-21T03:36:12.203 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-21T03:36:12.203 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-21T03:36:12.203 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-21T03:36:12.380 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-21T03:36:12.380 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-21T03:36:12.380 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-21T03:36:12.380 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-21T03:36:12.380 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-21T03:36:12.380 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-21T03:36:12.905 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-21T03:36:12.905 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:12.906 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:36:12.906 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-21T03:36:12.906 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-21T03:36:12.906 INFO:teuthology.orchestra.run.smithi148.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-21T03:36:12.906 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:12.906 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:36:13.113 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph 2023-12-21T03:36:13.132 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-21T03:36:13.133 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:13.133 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:36:13.133 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-21T03:36:13.133 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-21T03:36:13.133 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-21T03:36:13.133 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:13.133 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:36:13.384 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph 2023-12-21T03:36:13.975 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:54 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:36:14.242 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:54 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:36:15.703 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-21T03:36:15.707 INFO:teuthology.orchestra.run.smithi148.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-21T03:36:15.708 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout:Install 32 Packages 2023-12-21T03:36:15.709 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:15.710 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 31 M 2023-12-21T03:36:15.710 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 111 M 2023-12-21T03:36:15.710 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:36:15.936 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:36:15.940 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:15.940 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-12-21T03:36:15.940 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:15.940 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:36:15.940 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-21T03:36:15.940 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-12-21T03:36:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-21T03:36:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-21T03:36:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-21T03:36:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-21T03:36:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-21T03:36:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-21T03:36:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-21T03:36:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-21T03:36:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:15.944 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:36:15.944 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:15.944 INFO:teuthology.orchestra.run.smithi078.stdout:Install 32 Packages 2023-12-21T03:36:15.944 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:15.944 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 31 M 2023-12-21T03:36:15.945 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 111 M 2023-12-21T03:36:15.945 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:36:16.353 INFO:teuthology.orchestra.run.smithi148.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 113 kB/s | 6.3 kB 00:00 2023-12-21T03:36:16.403 INFO:teuthology.orchestra.run.smithi148.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-12-21T03:36:16.462 INFO:teuthology.orchestra.run.smithi148.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2023-12-21T03:36:16.487 INFO:teuthology.orchestra.run.smithi148.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2023-12-21T03:36:16.573 INFO:teuthology.orchestra.run.smithi078.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 104 kB/s | 6.3 kB 00:00 2023-12-21T03:36:16.597 INFO:teuthology.orchestra.run.smithi148.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2023-12-21T03:36:16.632 INFO:teuthology.orchestra.run.smithi078.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2023-12-21T03:36:16.682 INFO:teuthology.orchestra.run.smithi078.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2023-12-21T03:36:16.707 INFO:teuthology.orchestra.run.smithi078.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2023-12-21T03:36:16.791 INFO:teuthology.orchestra.run.smithi078.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2023-12-21T03:36:16.879 INFO:teuthology.orchestra.run.smithi148.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 38 MB/s | 18 MB 00:00 2023-12-21T03:36:16.904 INFO:teuthology.orchestra.run.smithi148.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 126 kB/s | 39 kB 00:00 2023-12-21T03:36:16.929 INFO:teuthology.orchestra.run.smithi148.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 355 kB/s | 157 kB 00:00 2023-12-21T03:36:17.005 INFO:teuthology.orchestra.run.smithi148.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 568 kB/s | 57 kB 00:00 2023-12-21T03:36:17.033 INFO:teuthology.orchestra.run.smithi078.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 45 MB/s | 18 MB 00:00 2023-12-21T03:36:17.058 INFO:teuthology.orchestra.run.smithi078.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 145 kB/s | 39 kB 00:00 2023-12-21T03:36:17.072 INFO:teuthology.orchestra.run.smithi148.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.1 MB/s | 457 kB 00:00 2023-12-21T03:36:17.108 INFO:teuthology.orchestra.run.smithi078.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 391 kB/s | 157 kB 00:00 2023-12-21T03:36:17.114 INFO:teuthology.orchestra.run.smithi148.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 442 kB/s | 103 kB 00:00 2023-12-21T03:36:17.159 INFO:teuthology.orchestra.run.smithi078.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 568 kB/s | 57 kB 00:00 2023-12-21T03:36:17.222 INFO:teuthology.orchestra.run.smithi148.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 249 kB 00:00 2023-12-21T03:36:17.289 INFO:teuthology.orchestra.run.smithi148.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 622 kB/s | 109 kB 00:00 2023-12-21T03:36:17.292 INFO:teuthology.orchestra.run.smithi078.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 399 kB/s | 103 kB 00:00 2023-12-21T03:36:17.318 INFO:teuthology.orchestra.run.smithi078.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.1 MB/s | 457 kB 00:00 2023-12-21T03:36:17.331 INFO:teuthology.orchestra.run.smithi148.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 2.4 MB/s | 641 kB 00:00 2023-12-21T03:36:17.376 INFO:teuthology.orchestra.run.smithi078.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 249 kB 00:00 2023-12-21T03:36:17.381 INFO:teuthology.orchestra.run.smithi148.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 275 kB/s | 44 kB 00:00 2023-12-21T03:36:17.482 INFO:teuthology.orchestra.run.smithi148.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 4.0 MB/s | 384 kB 00:00 2023-12-21T03:36:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 622 kB/s | 109 kB 00:00 2023-12-21T03:36:17.532 INFO:teuthology.orchestra.run.smithi148.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 765 kB/s | 185 kB 00:00 2023-12-21T03:36:17.558 INFO:teuthology.orchestra.run.smithi148.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2023-12-21T03:36:17.574 INFO:teuthology.orchestra.run.smithi148.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 418 kB/s | 18 kB 00:00 2023-12-21T03:36:17.579 INFO:teuthology.orchestra.run.smithi078.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 2.2 MB/s | 641 kB 00:00 2023-12-21T03:36:17.600 INFO:teuthology.orchestra.run.smithi148.stdout:(20/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 657 kB/s | 173 kB 00:00 2023-12-21T03:36:17.602 INFO:teuthology.orchestra.run.smithi078.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 193 kB/s | 44 kB 00:00 2023-12-21T03:36:17.625 INFO:teuthology.orchestra.run.smithi148.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 726 kB/s | 49 kB 00:00 2023-12-21T03:36:17.642 INFO:teuthology.orchestra.run.smithi148.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 875 kB/s | 59 kB 00:00 2023-12-21T03:36:17.676 INFO:teuthology.orchestra.run.smithi148.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.7 MB/s | 283 kB 00:00 2023-12-21T03:36:17.692 INFO:teuthology.orchestra.run.smithi148.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 232 kB/s | 16 kB 00:00 2023-12-21T03:36:17.709 INFO:teuthology.orchestra.run.smithi148.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 279 kB/s | 19 kB 00:00 2023-12-21T03:36:17.711 INFO:teuthology.orchestra.run.smithi078.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-12-21T03:36:17.726 INFO:teuthology.orchestra.run.smithi148.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2023-12-21T03:36:17.752 INFO:teuthology.orchestra.run.smithi148.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 661 kB/s | 27 kB 00:00 2023-12-21T03:36:17.769 INFO:teuthology.orchestra.run.smithi078.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 671 kB/s | 185 kB 00:00 2023-12-21T03:36:17.777 INFO:teuthology.orchestra.run.smithi148.stdout:(28/32): python3-tempora-1.14.1-5.el8.noarch.rp 437 kB/s | 29 kB 00:00 2023-12-21T03:36:17.786 INFO:teuthology.orchestra.run.smithi078.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 147 kB/s | 11 kB 00:00 2023-12-21T03:36:17.802 INFO:teuthology.orchestra.run.smithi148.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-12-21T03:36:17.820 INFO:teuthology.orchestra.run.smithi078.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2023-12-21T03:36:17.852 INFO:teuthology.orchestra.run.smithi148.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.4 MB/s | 251 kB 00:00 2023-12-21T03:36:17.853 INFO:teuthology.orchestra.run.smithi078.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2023-12-21T03:36:17.870 INFO:teuthology.orchestra.run.smithi078.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 581 kB/s | 49 kB 00:00 2023-12-21T03:36:17.878 INFO:teuthology.orchestra.run.smithi148.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2023-12-21T03:36:17.887 INFO:teuthology.orchestra.run.smithi078.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 873 kB/s | 59 kB 00:00 2023-12-21T03:36:17.903 INFO:teuthology.orchestra.run.smithi148.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 680 kB/s | 85 kB 00:00 2023-12-21T03:36:17.907 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:17.907 INFO:teuthology.orchestra.run.smithi148.stdout:Total 14 MB/s | 31 MB 00:02 2023-12-21T03:36:17.921 INFO:teuthology.orchestra.run.smithi078.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2023-12-21T03:36:17.946 INFO:teuthology.orchestra.run.smithi078.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-12-21T03:36:17.971 INFO:teuthology.orchestra.run.smithi078.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 222 kB/s | 19 kB 00:00 2023-12-21T03:36:17.992 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:36:17.997 INFO:teuthology.orchestra.run.smithi078.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2023-12-21T03:36:18.030 INFO:teuthology.orchestra.run.smithi078.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 349 kB/s | 29 kB 00:00 2023-12-21T03:36:18.032 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:36:18.032 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:36:18.081 INFO:teuthology.orchestra.run.smithi078.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2023-12-21T03:36:18.106 INFO:teuthology.orchestra.run.smithi078.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2023-12-21T03:36:18.131 INFO:teuthology.orchestra.run.smithi078.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.4 MB/s | 251 kB 00:00 2023-12-21T03:36:18.157 INFO:teuthology.orchestra.run.smithi078.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-12-21T03:36:18.182 INFO:teuthology.orchestra.run.smithi078.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 304 kB/s | 23 kB 00:00 2023-12-21T03:36:18.184 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:18.184 INFO:teuthology.orchestra.run.smithi078.stdout:Total 14 MB/s | 31 MB 00:02 2023-12-21T03:36:18.267 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:36:18.306 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:36:18.306 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:36:18.349 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:36:18.352 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:36:18.627 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:36:18.630 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:36:19.527 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:36:19.736 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-21T03:36:19.736 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:36:19.937 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-21T03:36:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-21T03:36:20.175 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-21T03:36:20.213 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-21T03:36:20.418 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-21T03:36:20.470 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-21T03:36:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-21T03:36:20.688 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-21T03:36:20.806 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-21T03:36:20.889 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-21T03:36:20.994 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-21T03:36:21.091 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-21T03:36:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-21T03:36:21.260 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-21T03:36:21.382 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-21T03:36:21.486 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-21T03:36:21.600 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-21T03:36:21.741 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-21T03:36:21.794 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-21T03:36:21.950 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-21T03:36:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-21T03:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-21T03:36:22.298 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-21T03:36:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-21T03:36:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-21T03:36:22.841 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-21T03:36:22.895 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-21T03:36:23.046 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-21T03:36:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-21T03:36:23.329 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-21T03:36:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-21T03:36:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-21T03:36:23.560 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-21T03:36:23.839 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-21T03:36:23.844 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-21T03:36:24.077 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-21T03:36:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-21T03:36:24.261 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-21T03:36:24.282 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-21T03:36:24.504 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-21T03:36:24.508 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-21T03:36:24.750 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-21T03:36:24.752 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-21T03:36:24.963 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-21T03:36:25.001 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-21T03:36:25.226 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-21T03:36:25.242 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-21T03:36:25.553 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-21T03:36:25.610 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-21T03:36:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-21T03:36:25.943 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-21T03:36:26.249 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-21T03:36:26.343 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-21T03:36:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-21T03:36:26.564 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-21T03:36:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-21T03:36:28.095 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-21T03:36:28.162 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-21T03:36:28.322 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-21T03:36:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-21T03:36:28.756 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-21T03:36:28.804 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-21T03:36:29.022 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-21T03:36:29.027 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-21T03:36:29.201 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-21T03:36:29.219 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-21T03:36:29.246 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-21T03:36:29.358 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-21T03:36:29.438 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-21T03:36:29.455 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-21T03:36:29.612 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-21T03:36:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-21T03:36:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-21T03:36:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-21T03:36:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-21T03:36:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-21T03:36:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-21T03:36:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-21T03:36:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-21T03:36:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-21T03:36:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-21T03:36:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-21T03:36:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-21T03:36:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-21T03:36:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-21T03:36:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-21T03:36:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-21T03:36:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-21T03:36:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-21T03:36:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-21T03:36:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-21T03:36:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-21T03:36:30.031 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-21T03:36:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-21T03:36:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-21T03:36:30.034 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-21T03:36:30.034 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-21T03:36:30.034 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-21T03:36:30.977 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-21T03:36:30.978 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-2.6-1.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-21T03:36:30.979 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-21T03:36:30.980 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-21T03:36:30.980 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-21T03:36:30.980 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-21T03:36:30.980 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-21T03:36:30.980 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-21T03:36:30.980 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-21T03:36:30.980 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:30.980 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:36:31.208 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-base 2023-12-21T03:36:31.295 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-21T03:36:31.295 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:31.295 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:36:31.295 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:31.295 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:31.295 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-21T03:36:31.296 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend-2.6-1.el8.noarch 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-21T03:36:31.297 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-21T03:36:31.298 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-21T03:36:31.298 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-21T03:36:31.298 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-21T03:36:31.298 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:31.298 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:36:31.497 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-base 2023-12-21T03:36:32.064 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:12 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:36:32.370 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:13 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:36:33.498 INFO:teuthology.orchestra.run.smithi078.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:36:33.724 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:36:33.725 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-12-21T03:36:33.726 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:36:33.827 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install cephadm 2023-12-21T03:36:33.836 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:36:34.066 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:36:34.067 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-12-21T03:36:34.068 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:36:34.180 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install cephadm 2023-12-21T03:36:34.676 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:14 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:36:35.040 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:16 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:36:36.359 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:36:36.360 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-12-21T03:36:36.360 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:36.360 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:36:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-21T03:36:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:36.361 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:36:36.361 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:36.361 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2023-12-21T03:36:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:36.361 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 71 k 2023-12-21T03:36:36.361 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 290 k 2023-12-21T03:36:36.361 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:36:36.412 INFO:teuthology.orchestra.run.smithi078.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2023-12-21T03:36:36.413 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:36.413 INFO:teuthology.orchestra.run.smithi078.stdout:Total 1.3 MB/s | 71 kB 00:00 2023-12-21T03:36:36.413 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:36:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:36:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:36:36.421 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:36:36.425 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:36:36.468 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:36:36.743 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:36:36.744 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:36.744 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-12-21T03:36:36.744 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:36.744 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:36:36.744 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-21T03:36:36.744 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:36.744 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:36:36.744 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:36.744 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-12-21T03:36:36.744 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:36.745 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 71 k 2023-12-21T03:36:36.745 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 290 k 2023-12-21T03:36:36.745 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:36:36.791 INFO:teuthology.orchestra.run.smithi148.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2023-12-21T03:36:36.792 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:36.792 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.5 MB/s | 71 kB 00:00 2023-12-21T03:36:36.792 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:36:36.799 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:36:36.799 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:36:36.801 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:36:36.804 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:36:36.875 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:36:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-21T03:36:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-21T03:36:37.352 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-21T03:36:37.381 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-21T03:36:37.591 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-21T03:36:37.816 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-21T03:36:38.035 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-21T03:36:38.036 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:36:38.036 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-21T03:36:38.036 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:36:38.242 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-immutable-object-cache 2023-12-21T03:36:39.064 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-21T03:36:39.064 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:39.064 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:36:39.064 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-21T03:36:39.064 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:39.064 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:36:39.106 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:19 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:36:39.312 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-immutable-object-cache 2023-12-21T03:36:40.179 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:21 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:36:40.781 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:36:40.781 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:40.781 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repo Size 2023-12-21T03:36:40.782 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:40.782 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:36:40.782 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-21T03:36:40.782 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:40.782 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:36:40.782 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:40.782 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2023-12-21T03:36:40.782 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:40.783 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 163 k 2023-12-21T03:36:40.783 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 587 k 2023-12-21T03:36:40.783 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:36:40.859 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 2.1 MB/s | 163 kB 00:00 2023-12-21T03:36:40.859 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:40.859 INFO:teuthology.orchestra.run.smithi078.stdout:Total 2.1 MB/s | 163 kB 00:00 2023-12-21T03:36:40.860 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:36:40.870 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:36:40.870 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:36:40.917 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:36:40.920 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:36:41.033 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:36:41.207 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:36:41.629 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:36:41.875 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:36:41.875 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:41.875 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-12-21T03:36:41.875 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:41.875 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:36:41.876 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-21T03:36:41.876 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:41.876 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:36:41.876 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:41.876 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-12-21T03:36:41.876 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:41.876 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 163 k 2023-12-21T03:36:41.876 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 587 k 2023-12-21T03:36:41.877 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:36:41.929 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.0 MB/s | 163 kB 00:00 2023-12-21T03:36:41.929 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:41.930 INFO:teuthology.orchestra.run.smithi148.stdout:Total 3.0 MB/s | 163 kB 00:00 2023-12-21T03:36:41.930 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:36:41.940 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:36:41.941 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:36:41.987 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:36:41.990 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:36:42.122 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:36:42.310 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:36:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:36:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:42.312 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:36:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:42.312 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:36:42.529 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr 2023-12-21T03:36:42.731 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:36:43.383 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:23 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:36:43.404 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:36:43.404 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:43.404 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:36:43.404 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-21T03:36:43.404 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:43.404 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:36:43.610 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr 2023-12-21T03:36:44.473 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:25 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:36:44.833 INFO:teuthology.orchestra.run.smithi078.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:36:45.064 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:36:45.065 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-12-21T03:36:45.065 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:36:45.602 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-dashboard 2023-12-21T03:36:45.944 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:36:46.171 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:36:46.172 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-12-21T03:36:46.172 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:36:46.285 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-dashboard 2023-12-21T03:36:46.456 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:26 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:36:47.147 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:28 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:36:48.156 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:36:48.158 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:48.158 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-12-21T03:36:48.158 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:48.158 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:36:48.158 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-21T03:36:48.158 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-12-21T03:36:48.158 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-21T03:36:48.158 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-21T03:36:48.159 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-21T03:36:48.159 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-21T03:36:48.159 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-21T03:36:48.159 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:48.159 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:36:48.159 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:36:48.159 INFO:teuthology.orchestra.run.smithi078.stdout:Install 6 Packages 2023-12-21T03:36:48.159 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:48.160 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 3.0 M 2023-12-21T03:36:48.160 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 15 M 2023-12-21T03:36:48.160 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:36:48.737 INFO:teuthology.orchestra.run.smithi078.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 465 kB/s | 24 kB 00:00 2023-12-21T03:36:48.762 INFO:teuthology.orchestra.run.smithi078.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 116 kB/s | 8.9 kB 00:00 2023-12-21T03:36:48.842 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:36:48.843 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:48.844 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-12-21T03:36:48.844 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:48.844 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:36:48.844 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-21T03:36:48.844 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-12-21T03:36:48.844 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-21T03:36:48.845 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-21T03:36:48.845 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-21T03:36:48.845 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-21T03:36:48.845 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-21T03:36:48.845 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:48.845 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:36:48.845 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:36:48.846 INFO:teuthology.orchestra.run.smithi148.stdout:Install 6 Packages 2023-12-21T03:36:48.846 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:48.846 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 3.0 M 2023-12-21T03:36:48.846 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 15 M 2023-12-21T03:36:48.847 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:36:48.863 INFO:teuthology.orchestra.run.smithi078.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 15 MB/s | 2.7 MB 00:00 2023-12-21T03:36:48.980 INFO:teuthology.orchestra.run.smithi078.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 176 kB/s | 43 kB 00:00 2023-12-21T03:36:49.080 INFO:teuthology.orchestra.run.smithi078.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 106 kB/s | 34 kB 00:00 2023-12-21T03:36:49.238 INFO:teuthology.orchestra.run.smithi078.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 524 kB/s | 196 kB 00:00 2023-12-21T03:36:49.240 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:49.241 INFO:teuthology.orchestra.run.smithi078.stdout:Total 2.7 MB/s | 3.0 MB 00:01 2023-12-21T03:36:49.252 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:36:49.262 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:36:49.262 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:36:49.433 INFO:teuthology.orchestra.run.smithi148.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 489 kB/s | 24 kB 00:00 2023-12-21T03:36:49.453 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:36:49.456 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:36:49.458 INFO:teuthology.orchestra.run.smithi148.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 120 kB/s | 8.9 kB 00:00 2023-12-21T03:36:49.509 INFO:teuthology.orchestra.run.smithi148.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 21 MB/s | 2.7 MB 00:00 2023-12-21T03:36:49.592 INFO:teuthology.orchestra.run.smithi148.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 252 kB/s | 34 kB 00:00 2023-12-21T03:36:49.618 INFO:teuthology.orchestra.run.smithi148.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 231 kB/s | 43 kB 00:00 2023-12-21T03:36:49.685 INFO:teuthology.orchestra.run.smithi148.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2023-12-21T03:36:49.687 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:49.687 INFO:teuthology.orchestra.run.smithi148.stdout:Total 3.5 MB/s | 3.0 MB 00:00 2023-12-21T03:36:49.698 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:36:49.707 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:36:49.708 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:36:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:36:49.877 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:36:49.880 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:36:49.962 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-21T03:36:50.176 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:36:50.223 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-21T03:36:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-21T03:36:50.449 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-21T03:36:50.571 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-21T03:36:50.736 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-21T03:36:50.961 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-21T03:36:51.065 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-21T03:36:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-21T03:36:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-21T03:36:52.070 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-21T03:36:52.112 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-21T03:36:52.113 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-21T03:36:52.113 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-21T03:36:52.113 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-21T03:36:52.113 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-21T03:36:52.113 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-21T03:36:52.326 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-21T03:36:52.617 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-21T03:36:52.617 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-21T03:36:52.618 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-21T03:36:52.618 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-21T03:36:52.618 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-21T03:36:52.618 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-21T03:36:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-21T03:36:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:36:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-21T03:36:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-21T03:36:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-21T03:36:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-21T03:36:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-21T03:36:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-21T03:36:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:36:52.923 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-21T03:36:53.375 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-21T03:36:53.375 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:53.375 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:36:53.375 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-21T03:36:53.375 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-21T03:36:53.375 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-21T03:36:53.375 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-21T03:36:53.375 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-21T03:36:53.376 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-21T03:36:53.376 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:53.376 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:36:53.500 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-21T03:36:53.774 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:33 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:36:54.361 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:35 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:36:55.474 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:36:55.475 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2023-12-21T03:36:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-12-21T03:36:55.475 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2023-12-21T03:36:55.475 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:36:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-21T03:36:55.475 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-12-21T03:36:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-21T03:36:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-21T03:36:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-21T03:36:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-21T03:36:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-21T03:36:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-21T03:36:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2023-12-21T03:36:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:55.476 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:36:55.476 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2023-12-21T03:36:55.476 INFO:teuthology.orchestra.run.smithi078.stdout:Install 8 Packages 2023-12-21T03:36:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:36:55.476 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 37 M 2023-12-21T03:36:55.477 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 210 M 2023-12-21T03:36:55.477 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:36:55.728 INFO:teuthology.orchestra.run.smithi078.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 51 MB/s | 8.6 MB 00:00 2023-12-21T03:36:56.047 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:36:56.048 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================================ 2023-12-21T03:36:56.048 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-12-21T03:36:56.048 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================================ 2023-12-21T03:36:56.048 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:36:56.048 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-21T03:36:56.048 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-12-21T03:36:56.048 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-21T03:36:56.048 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-21T03:36:56.048 INFO:teuthology.orchestra.run.smithi148.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-21T03:36:56.049 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-21T03:36:56.049 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-21T03:36:56.049 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-21T03:36:56.049 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2023-12-21T03:36:56.049 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:56.049 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:36:56.049 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================================ 2023-12-21T03:36:56.049 INFO:teuthology.orchestra.run.smithi148.stdout:Install 8 Packages 2023-12-21T03:36:56.049 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:36:56.050 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 37 M 2023-12-21T03:36:56.050 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 210 M 2023-12-21T03:36:56.051 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:36:56.291 INFO:teuthology.orchestra.run.smithi148.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 60 MB/s | 8.6 MB 00:00 2023-12-21T03:36:56.465 INFO:teuthology.orchestra.run.smithi078.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.1 MB/s | 3.7 MB 00:00 2023-12-21T03:36:56.531 INFO:teuthology.orchestra.run.smithi078.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 4.9 MB/s | 4.7 MB 00:00 2023-12-21T03:36:56.556 INFO:teuthology.orchestra.run.smithi078.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-12-21T03:36:56.615 INFO:teuthology.orchestra.run.smithi078.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.6 MB/s | 4.9 MB 00:01 2023-12-21T03:36:56.824 INFO:teuthology.orchestra.run.smithi078.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 827 kB/s | 172 kB 00:00 2023-12-21T03:36:56.858 INFO:teuthology.orchestra.run.smithi148.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 6.7 MB/s | 4.7 MB 00:00 2023-12-21T03:36:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.9 MB/s | 225 kB 00:00 2023-12-21T03:36:57.124 INFO:teuthology.orchestra.run.smithi078.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2023-12-21T03:36:57.176 INFO:teuthology.orchestra.run.smithi148.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.7 MB/s | 4.9 MB 00:01 2023-12-21T03:36:57.341 INFO:teuthology.orchestra.run.smithi078.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 18 MB/s | 14 MB 00:00 2023-12-21T03:36:57.342 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:57.342 INFO:teuthology.orchestra.run.smithi078.stdout:Total 20 MB/s | 37 MB 00:01 2023-12-21T03:36:57.442 INFO:teuthology.orchestra.run.smithi148.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.2 MB/s | 3.7 MB 00:01 2023-12-21T03:36:57.468 INFO:teuthology.orchestra.run.smithi148.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.2 MB/s | 645 kB 00:00 2023-12-21T03:36:57.575 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:36:57.593 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:36:57.593 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:36:57.726 INFO:teuthology.orchestra.run.smithi148.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 18 MB/s | 14 MB 00:00 2023-12-21T03:36:57.752 INFO:teuthology.orchestra.run.smithi148.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 559 kB/s | 172 kB 00:00 2023-12-21T03:36:57.752 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:36:57.753 INFO:teuthology.orchestra.run.smithi148.stdout:Total 22 MB/s | 37 MB 00:01 2023-12-21T03:36:57.921 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:36:57.924 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:36:57.986 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:36:58.004 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:36:58.004 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:36:58.359 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:36:58.362 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:36:58.546 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:36:58.705 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-21T03:36:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-21T03:36:58.988 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-21T03:36:58.989 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:36:59.141 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-21T03:36:59.272 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-21T03:36:59.391 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-21T03:36:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-21T03:36:59.780 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-21T03:37:00.009 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-21T03:37:00.166 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-21T03:37:00.402 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-21T03:37:00.529 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-21T03:37:00.813 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-21T03:37:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-21T03:37:02.057 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-21T03:37:02.622 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-21T03:37:02.757 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-21T03:37:02.873 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-21T03:37:03.110 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-21T03:37:03.415 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-21T03:37:04.657 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-21T03:37:05.183 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-21T03:37:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2023-12-21T03:37:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-21T03:37:06.482 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-21T03:37:06.482 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-21T03:37:06.483 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-21T03:37:06.483 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-21T03:37:06.483 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-21T03:37:06.483 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-21T03:37:06.483 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2023-12-21T03:37:06.483 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-21T03:37:06.524 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2023-12-21T03:37:06.783 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-21T03:37:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-21T03:37:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-21T03:37:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-21T03:37:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-21T03:37:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-21T03:37:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-21T03:37:07.092 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2023-12-21T03:37:07.092 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-21T03:37:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-21T03:37:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:07.253 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:37:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-21T03:37:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-21T03:37:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-21T03:37:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-21T03:37:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-21T03:37:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-21T03:37:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-21T03:37:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2023-12-21T03:37:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:07.254 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:07.376 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-rook 2023-12-21T03:37:07.867 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-21T03:37:07.867 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:07.868 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:37:07.868 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-21T03:37:07.868 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-21T03:37:07.868 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-21T03:37:07.868 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-21T03:37:07.868 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-21T03:37:07.868 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-21T03:37:07.868 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-21T03:37:07.868 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2023-12-21T03:37:07.868 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:07.868 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:07.991 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-rook 2023-12-21T03:37:08.229 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:48 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:08.848 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:49 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:09.934 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:09.935 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:09.935 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-12-21T03:37:09.935 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-21T03:37:09.936 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-21T03:37:09.937 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:09.937 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:37:09.937 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:09.937 INFO:teuthology.orchestra.run.smithi078.stdout:Install 13 Packages 2023-12-21T03:37:09.937 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:09.937 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 1.9 M 2023-12-21T03:37:09.937 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 24 M 2023-12-21T03:37:09.938 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:37:10.550 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:10.552 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:10.552 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-12-21T03:37:10.552 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:10.552 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:37:10.552 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-21T03:37:10.552 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-12-21T03:37:10.552 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-21T03:37:10.552 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:10.553 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:37:10.554 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:10.554 INFO:teuthology.orchestra.run.smithi148.stdout:Install 13 Packages 2023-12-21T03:37:10.554 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:10.554 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 1.9 M 2023-12-21T03:37:10.554 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 24 M 2023-12-21T03:37:10.554 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:37:11.477 INFO:teuthology.orchestra.run.smithi148.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 632 kB/s | 27 kB 00:00 2023-12-21T03:37:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 740 kB/s | 27 kB 00:00 2023-12-21T03:37:11.577 INFO:teuthology.orchestra.run.smithi148.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 140 kB/s | 20 kB 00:00 2023-12-21T03:37:11.602 INFO:teuthology.orchestra.run.smithi148.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 158 kB/s | 27 kB 00:00 2023-12-21T03:37:11.624 INFO:teuthology.orchestra.run.smithi078.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 183 kB/s | 27 kB 00:00 2023-12-21T03:37:11.649 INFO:teuthology.orchestra.run.smithi078.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 117 kB/s | 20 kB 00:00 2023-12-21T03:37:11.694 INFO:teuthology.orchestra.run.smithi148.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 582 kB/s | 126 kB 00:00 2023-12-21T03:37:11.720 INFO:teuthology.orchestra.run.smithi148.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 773 kB/s | 110 kB 00:00 2023-12-21T03:37:11.741 INFO:teuthology.orchestra.run.smithi078.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 561 kB/s | 126 kB 00:00 2023-12-21T03:37:11.766 INFO:teuthology.orchestra.run.smithi078.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 774 kB/s | 110 kB 00:00 2023-12-21T03:37:11.770 INFO:teuthology.orchestra.run.smithi148.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 926 kB/s | 155 kB 00:00 2023-12-21T03:37:11.821 INFO:teuthology.orchestra.run.smithi148.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 328 kB/s | 33 kB 00:00 2023-12-21T03:37:11.825 INFO:teuthology.orchestra.run.smithi078.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 883 kB/s | 155 kB 00:00 2023-12-21T03:37:11.846 INFO:teuthology.orchestra.run.smithi148.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 282 kB/s | 43 kB 00:00 2023-12-21T03:37:11.871 INFO:teuthology.orchestra.run.smithi148.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 161 kB/s | 16 kB 00:00 2023-12-21T03:37:11.875 INFO:teuthology.orchestra.run.smithi078.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 303 kB/s | 33 kB 00:00 2023-12-21T03:37:11.888 INFO:teuthology.orchestra.run.smithi148.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 1.2 MB/s | 81 kB 00:00 2023-12-21T03:37:11.918 INFO:teuthology.orchestra.run.smithi078.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 242 kB/s | 43 kB 00:00 2023-12-21T03:37:11.943 INFO:teuthology.orchestra.run.smithi078.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 139 kB/s | 16 kB 00:00 2023-12-21T03:37:11.967 INFO:teuthology.orchestra.run.smithi148.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 778 kB/s | 61 kB 00:00 2023-12-21T03:37:11.993 INFO:teuthology.orchestra.run.smithi078.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 695 kB/s | 81 kB 00:00 2023-12-21T03:37:11.997 INFO:teuthology.orchestra.run.smithi148.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 513 kB/s | 64 kB 00:00 2023-12-21T03:37:12.043 INFO:teuthology.orchestra.run.smithi078.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-12-21T03:37:12.072 INFO:teuthology.orchestra.run.smithi148.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.2 MB/s | 1.2 MB 00:00 2023-12-21T03:37:12.073 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:37:12.073 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-12-21T03:37:12.102 INFO:teuthology.orchestra.run.smithi078.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 405 kB/s | 64 kB 00:00 2023-12-21T03:37:12.132 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:37:12.135 INFO:teuthology.orchestra.run.smithi078.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.4 MB/s | 1.2 MB 00:00 2023-12-21T03:37:12.135 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:37:12.136 INFO:teuthology.orchestra.run.smithi078.stdout:Total 890 kB/s | 1.9 MB 00:02 2023-12-21T03:37:12.148 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:37:12.148 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:37:12.193 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:37:12.208 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:37:12.209 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:37:12.234 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:37:12.237 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:37:12.293 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:37:12.296 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:37:12.807 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:37:12.809 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:37:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-21T03:37:13.092 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-21T03:37:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-21T03:37:13.322 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-21T03:37:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-21T03:37:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-21T03:37:13.743 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-21T03:37:13.812 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-21T03:37:13.978 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-21T03:37:14.041 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-21T03:37:14.206 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-21T03:37:14.286 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-21T03:37:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-21T03:37:14.622 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-21T03:37:14.732 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-21T03:37:14.920 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-21T03:37:15.785 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-21T03:37:15.960 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-21T03:37:16.067 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-21T03:37:16.245 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-21T03:37:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-21T03:37:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-21T03:37:16.479 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-21T03:37:16.644 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-21T03:37:16.662 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-21T03:37:16.871 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-21T03:37:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-21T03:37:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-21T03:37:17.180 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-21T03:37:17.180 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-21T03:37:17.180 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-21T03:37:17.180 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-21T03:37:17.180 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-21T03:37:17.181 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-21T03:37:17.181 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-21T03:37:17.181 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-21T03:37:17.181 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-21T03:37:17.181 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-21T03:37:17.181 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-21T03:37:17.181 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-21T03:37:17.181 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-21T03:37:17.684 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-21T03:37:17.684 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:17.684 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:37:17.684 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:17.685 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:17.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-cephadm 2023-12-21T03:37:17.917 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-21T03:37:17.917 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:17.917 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-21T03:37:17.918 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:17.919 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:18.026 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-cephadm 2023-12-21T03:37:18.663 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:58 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:18.884 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:59 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:20.350 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:20.350 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:20.351 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-12-21T03:37:20.351 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:20.351 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:37:20.351 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-21T03:37:20.351 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-12-21T03:37:20.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-21T03:37:20.352 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-21T03:37:20.352 INFO:teuthology.orchestra.run.smithi078.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-21T03:37:20.352 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-21T03:37:20.352 INFO:teuthology.orchestra.run.smithi078.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-21T03:37:20.352 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:20.352 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:37:20.352 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:20.352 INFO:teuthology.orchestra.run.smithi078.stdout:Install 6 Packages 2023-12-21T03:37:20.353 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:20.353 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 5.8 M 2023-12-21T03:37:20.353 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 24 M 2023-12-21T03:37:20.353 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:37:20.556 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:20.558 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:20.558 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-12-21T03:37:20.558 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:20.558 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:37:20.558 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-21T03:37:20.558 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-12-21T03:37:20.559 INFO:teuthology.orchestra.run.smithi148.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-21T03:37:20.559 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-21T03:37:20.559 INFO:teuthology.orchestra.run.smithi148.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-21T03:37:20.559 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-21T03:37:20.559 INFO:teuthology.orchestra.run.smithi148.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-21T03:37:20.559 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:20.560 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:37:20.560 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:20.560 INFO:teuthology.orchestra.run.smithi148.stdout:Install 6 Packages 2023-12-21T03:37:20.560 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:20.561 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 5.8 M 2023-12-21T03:37:20.561 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 24 M 2023-12-21T03:37:20.561 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:37:20.635 INFO:teuthology.orchestra.run.smithi078.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.3 MB/s | 89 kB 00:00 2023-12-21T03:37:20.910 INFO:teuthology.orchestra.run.smithi078.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.7 MB/s | 538 kB 00:00 2023-12-21T03:37:20.935 INFO:teuthology.orchestra.run.smithi078.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 71 kB/s | 21 kB 00:00 2023-12-21T03:37:20.955 INFO:teuthology.orchestra.run.smithi148.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 89 kB 00:00 2023-12-21T03:37:21.127 INFO:teuthology.orchestra.run.smithi078.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 9.0 MB/s | 4.8 MB 00:00 2023-12-21T03:37:21.163 INFO:teuthology.orchestra.run.smithi148.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 102 kB/s | 21 kB 00:00 2023-12-21T03:37:21.230 INFO:teuthology.orchestra.run.smithi148.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.6 MB/s | 538 kB 00:00 2023-12-21T03:37:21.330 INFO:teuthology.orchestra.run.smithi148.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.1 MB/s | 355 kB 00:00 2023-12-21T03:37:21.352 INFO:teuthology.orchestra.run.smithi078.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 117 kB/s | 49 kB 00:00 2023-12-21T03:37:21.472 INFO:teuthology.orchestra.run.smithi148.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 8.3 MB/s | 4.8 MB 00:00 2023-12-21T03:37:21.498 INFO:teuthology.orchestra.run.smithi148.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 183 kB/s | 49 kB 00:00 2023-12-21T03:37:21.500 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:37:21.500 INFO:teuthology.orchestra.run.smithi148.stdout:Total 6.2 MB/s | 5.8 MB 00:00 2023-12-21T03:37:21.553 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:37:21.562 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:37:21.563 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:37:21.661 INFO:teuthology.orchestra.run.smithi078.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 473 kB/s | 355 kB 00:00 2023-12-21T03:37:21.662 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:37:21.662 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:37:21.662 INFO:teuthology.orchestra.run.smithi078.stdout:Total 4.4 MB/s | 5.8 MB 00:01 2023-12-21T03:37:21.665 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:37:21.715 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:37:21.724 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:37:21.725 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:37:21.824 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:37:21.827 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:37:21.963 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:37:22.116 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:37:22.245 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-21T03:37:22.419 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-21T03:37:22.564 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-21T03:37:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-21T03:37:23.614 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-21T03:37:23.849 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-21T03:37:23.999 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-21T03:37:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-21T03:37:24.234 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-21T03:37:24.425 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-21T03:37:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-21T03:37:24.634 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-21T03:37:24.756 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-21T03:37:24.756 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-21T03:37:24.756 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-21T03:37:24.757 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-21T03:37:24.757 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-21T03:37:24.757 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-21T03:37:24.926 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-21T03:37:24.927 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-21T03:37:24.927 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-21T03:37:24.927 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-21T03:37:24.927 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-21T03:37:24.927 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-21T03:37:25.476 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-21T03:37:25.476 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:25.477 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:37:25.477 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-21T03:37:25.477 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-21T03:37:25.477 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-21T03:37:25.477 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:25.477 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:25.600 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-fuse 2023-12-21T03:37:25.635 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-21T03:37:25.635 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:25.635 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:37:25.635 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-21T03:37:25.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-21T03:37:25.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-21T03:37:25.636 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:25.636 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:25.771 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-fuse 2023-12-21T03:37:26.465 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:07 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:26.625 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:06 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:28.154 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:28.155 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:28.155 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-12-21T03:37:28.155 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:28.155 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:37:28.155 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-21T03:37:28.156 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-12-21T03:37:28.156 INFO:teuthology.orchestra.run.smithi148.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-21T03:37:28.156 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:28.156 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:37:28.156 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:28.156 INFO:teuthology.orchestra.run.smithi148.stdout:Install 2 Packages 2023-12-21T03:37:28.156 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:28.156 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 968 k 2023-12-21T03:37:28.156 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 3.2 M 2023-12-21T03:37:28.156 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:37:28.314 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:28.315 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:28.315 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-12-21T03:37:28.315 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:28.315 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:37:28.315 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-21T03:37:28.315 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-12-21T03:37:28.316 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-21T03:37:28.316 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:28.316 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:37:28.316 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:28.316 INFO:teuthology.orchestra.run.smithi078.stdout:Install 2 Packages 2023-12-21T03:37:28.316 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:28.316 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 968 k 2023-12-21T03:37:28.317 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 3.2 M 2023-12-21T03:37:28.317 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:37:28.414 INFO:teuthology.orchestra.run.smithi078.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2023-12-21T03:37:28.434 INFO:teuthology.orchestra.run.smithi148.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 10 MB/s | 885 kB 00:00 2023-12-21T03:37:28.556 INFO:teuthology.orchestra.run.smithi078.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 416 kB/s | 83 kB 00:00 2023-12-21T03:37:28.557 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:37:28.557 INFO:teuthology.orchestra.run.smithi078.stdout:Total 3.9 MB/s | 968 kB 00:00 2023-12-21T03:37:28.567 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:37:28.587 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:37:28.587 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:37:28.592 INFO:teuthology.orchestra.run.smithi148.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 342 kB/s | 83 kB 00:00 2023-12-21T03:37:28.593 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:37:28.593 INFO:teuthology.orchestra.run.smithi148.stdout:Total 2.2 MB/s | 968 kB 00:00 2023-12-21T03:37:28.603 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:37:28.624 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:37:28.624 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:37:28.673 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:37:28.676 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:37:28.711 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:37:28.714 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:37:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:37:28.900 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:37:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-21T03:37:29.228 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-21T03:37:29.326 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-21T03:37:29.453 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-21T03:37:29.708 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-21T03:37:29.708 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-21T03:37:29.850 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-21T03:37:29.850 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-21T03:37:30.442 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-21T03:37:30.442 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:30.442 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:37:30.442 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-21T03:37:30.442 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:30.442 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:30.550 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-21T03:37:30.550 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:30.550 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:37:30.550 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-21T03:37:30.550 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:30.550 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:30.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librados-devel 2023-12-21T03:37:30.768 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados-devel 2023-12-21T03:37:31.505 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:11 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:31.636 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:12 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:33.197 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:33.198 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:33.198 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-12-21T03:37:33.198 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:33.198 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:37:33.198 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-21T03:37:33.198 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:33.198 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:37:33.198 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:33.198 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2023-12-21T03:37:33.198 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:33.199 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 131 k 2023-12-21T03:37:33.199 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 439 k 2023-12-21T03:37:33.199 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:37:33.251 INFO:teuthology.orchestra.run.smithi078.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.5 MB/s | 131 kB 00:00 2023-12-21T03:37:33.252 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:37:33.252 INFO:teuthology.orchestra.run.smithi078.stdout:Total 2.4 MB/s | 131 kB 00:00 2023-12-21T03:37:33.252 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:37:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:37:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:37:33.306 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:37:33.309 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:37:33.326 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:33.327 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:33.327 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-12-21T03:37:33.327 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:33.327 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:37:33.327 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-21T03:37:33.328 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:33.328 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:37:33.328 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:33.328 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-12-21T03:37:33.328 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:33.329 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 131 k 2023-12-21T03:37:33.329 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 439 k 2023-12-21T03:37:33.329 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:37:33.388 INFO:teuthology.orchestra.run.smithi148.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 131 kB 00:00 2023-12-21T03:37:33.389 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:37:33.389 INFO:teuthology.orchestra.run.smithi148.stdout:Total 2.1 MB/s | 131 kB 00:00 2023-12-21T03:37:33.389 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:37:33.394 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:37:33.394 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:37:33.421 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:37:33.444 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:37:33.447 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:37:33.816 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:37:34.133 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:34.781 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:35.348 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:35.875 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-21T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:36.522 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install libcephfs2 2023-12-21T03:37:36.565 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:36.565 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:36.565 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:37:36.565 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-21T03:37:36.565 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:36.565 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:36.765 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs2 2023-12-21T03:37:37.381 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:17 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:37.623 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:18 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:38.840 INFO:teuthology.orchestra.run.smithi078.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:39.067 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:39.069 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-12-21T03:37:39.069 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:39.078 INFO:teuthology.orchestra.run.smithi148.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:39.183 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install libcephfs-devel 2023-12-21T03:37:39.304 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:39.305 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-12-21T03:37:39.305 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:39.395 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs-devel 2023-12-21T03:37:40.048 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:20 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:40.257 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:21 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:41.733 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:41.734 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:41.734 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-12-21T03:37:41.734 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:41.735 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:37:41.735 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-21T03:37:41.735 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:41.735 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:37:41.735 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:37:41.735 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2023-12-21T03:37:41.735 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:41.735 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 24 k 2023-12-21T03:37:41.735 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 94 k 2023-12-21T03:37:41.735 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:37:41.808 INFO:teuthology.orchestra.run.smithi078.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 331 kB/s | 24 kB 00:00 2023-12-21T03:37:41.809 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:37:41.809 INFO:teuthology.orchestra.run.smithi078.stdout:Total 325 kB/s | 24 kB 00:00 2023-12-21T03:37:41.809 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:37:41.813 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:37:41.813 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:37:41.815 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:37:41.818 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:37:41.871 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:37:41.946 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:41.947 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:41.947 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-12-21T03:37:41.947 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:41.947 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:37:41.947 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-21T03:37:41.947 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:41.947 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:37:41.947 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:37:41.947 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-12-21T03:37:41.947 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:41.948 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 24 k 2023-12-21T03:37:41.948 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 94 k 2023-12-21T03:37:41.948 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:37:42.011 INFO:teuthology.orchestra.run.smithi148.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 387 kB/s | 24 kB 00:00 2023-12-21T03:37:42.012 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:37:42.012 INFO:teuthology.orchestra.run.smithi148.stdout:Total 380 kB/s | 24 kB 00:00 2023-12-21T03:37:42.012 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:37:42.015 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:37:42.015 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:37:42.017 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:37:42.020 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:37:42.068 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:37:42.089 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:42.291 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:42.560 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:43.004 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:37:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-21T03:37:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:37:43.005 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:43.128 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librados2 2023-12-21T03:37:43.214 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:37:43.214 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:43.214 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:37:43.214 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-21T03:37:43.214 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:37:43.214 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:43.327 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados2 2023-12-21T03:37:43.979 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:23 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:44.202 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:25 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:45.430 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:45.656 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:45.657 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-12-21T03:37:45.657 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:45.673 INFO:teuthology.orchestra.run.smithi148.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:45.762 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librbd1 2023-12-21T03:37:45.903 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:45.904 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-12-21T03:37:45.904 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:46.006 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librbd1 2023-12-21T03:37:46.623 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:26 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:46.863 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:27 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:48.085 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:48.311 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:48.312 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-12-21T03:37:48.358 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:48.359 INFO:teuthology.orchestra.run.smithi148.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:48.415 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rados 2023-12-21T03:37:48.540 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:48.541 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-12-21T03:37:48.542 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:48.632 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rados 2023-12-21T03:37:49.283 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:29 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:49.495 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:30 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:50.813 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:50.935 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:51.036 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:51.038 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-12-21T03:37:51.038 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:51.163 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rgw 2023-12-21T03:37:51.165 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:51.166 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-12-21T03:37:51.166 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:51.281 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rgw 2023-12-21T03:37:52.019 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:32 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:52.139 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:33 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:53.474 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:53.606 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:53.700 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:53.701 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-12-21T03:37:53.701 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:53.802 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-cephfs 2023-12-21T03:37:53.832 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:53.833 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-12-21T03:37:53.834 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:53.924 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-cephfs 2023-12-21T03:37:54.659 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:34 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:54.789 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:35 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:56.115 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:56.244 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:56.347 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:56.349 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-12-21T03:37:56.349 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:56.461 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rbd 2023-12-21T03:37:56.471 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:56.472 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-12-21T03:37:56.472 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:56.571 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rbd 2023-12-21T03:37:57.318 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:37 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:37:57.433 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:38 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:37:58.766 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:58.896 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-21T03:37:58.994 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:37:58.995 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-12-21T03:37:58.995 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:37:59.108 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-fuse 2023-12-21T03:37:59.128 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:37:59.129 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-12-21T03:37:59.129 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:37:59.221 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-fuse 2023-12-21T03:37:59.963 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:39 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:38:00.081 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:41 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:38:01.648 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:38:01.648 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:38:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-12-21T03:38:01.649 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:38:01.649 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:38:01.649 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-21T03:38:01.649 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:01.649 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:38:01.649 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:38:01.649 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2023-12-21T03:38:01.649 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:01.650 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 89 k 2023-12-21T03:38:01.650 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 298 k 2023-12-21T03:38:01.650 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:38:01.699 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2023-12-21T03:38:01.699 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:38:01.699 INFO:teuthology.orchestra.run.smithi078.stdout:Total 1.7 MB/s | 89 kB 00:00 2023-12-21T03:38:01.700 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:38:01.709 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:38:01.709 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:38:01.758 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:38:01.761 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:38:01.779 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:38:01.780 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:38:01.780 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-12-21T03:38:01.780 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:38:01.780 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:38:01.780 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-21T03:38:01.780 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:01.780 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:38:01.780 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:38:01.780 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-12-21T03:38:01.780 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:01.781 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 89 k 2023-12-21T03:38:01.781 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 298 k 2023-12-21T03:38:01.781 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:38:01.825 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2023-12-21T03:38:01.826 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:38:01.826 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.9 MB/s | 89 kB 00:00 2023-12-21T03:38:01.826 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:38:01.836 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:38:01.836 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:38:01.870 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:38:01.885 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:38:01.888 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:38:01.992 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:38:02.054 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:02.172 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:02.387 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:02.495 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:03.080 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:03.080 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:03.080 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:38:03.081 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-21T03:38:03.081 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:03.081 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:38:03.247 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:03.247 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:03.248 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:38:03.248 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-21T03:38:03.248 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:03.248 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:38:03.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-mirror 2023-12-21T03:38:03.463 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-mirror 2023-12-21T03:38:04.159 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:44 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:38:04.330 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:45 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:38:05.858 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:38:05.858 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 3.7 M 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 18 M 2023-12-21T03:38:05.859 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:38:05.960 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2023-12-21T03:38:05.961 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:38:05.961 INFO:teuthology.orchestra.run.smithi078.stdout:Total 36 MB/s | 3.7 MB 00:00 2023-12-21T03:38:05.962 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:38:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:38:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:38:06.015 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:38:06.015 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:38:06.015 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-12-21T03:38:06.015 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:38:06.015 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:38:06.015 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-21T03:38:06.015 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:06.015 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:38:06.016 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:38:06.016 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-12-21T03:38:06.016 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:06.016 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 3.7 M 2023-12-21T03:38:06.016 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 18 M 2023-12-21T03:38:06.016 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:38:06.037 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:38:06.040 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:38:06.113 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 39 MB/s | 3.7 MB 00:00 2023-12-21T03:38:06.113 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:38:06.113 INFO:teuthology.orchestra.run.smithi148.stdout:Total 39 MB/s | 3.7 MB 00:00 2023-12-21T03:38:06.113 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:38:06.124 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:38:06.125 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:38:06.187 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:38:06.190 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:38:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:38:06.669 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:38:06.721 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:06.923 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:07.327 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:07.836 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:38:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-21T03:38:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:07.836 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:38:08.014 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:08.015 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:08.015 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:38:08.015 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-21T03:38:08.015 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:08.015 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:38:08.274 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-nbd 2023-12-21T03:38:08.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-nbd 2023-12-21T03:38:09.142 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:50 ago on Thu 21 Dec 2023 03:35:19 AM UTC. 2023-12-21T03:38:09.155 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:49 ago on Thu 21 Dec 2023 03:35:20 AM UTC. 2023-12-21T03:38:10.834 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-21T03:38:10.835 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:38:10.835 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-12-21T03:38:10.835 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:38:10.835 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-21T03:38:10.835 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-21T03:38:10.835 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:10.835 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-21T03:38:10.835 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-21T03:38:10.836 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-12-21T03:38:10.836 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:10.836 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 173 k 2023-12-21T03:38:10.836 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 649 k 2023-12-21T03:38:10.836 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-21T03:38:10.841 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-12-21T03:38:10.841 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:38:10.842 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-12-21T03:38:10.842 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:38:10.842 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-12-21T03:38:10.842 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-21T03:38:10.842 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:10.842 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-12-21T03:38:10.842 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-12-21T03:38:10.842 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2023-12-21T03:38:10.842 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:10.843 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 173 k 2023-12-21T03:38:10.843 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 649 k 2023-12-21T03:38:10.843 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-12-21T03:38:10.880 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.6 MB/s | 173 kB 00:00 2023-12-21T03:38:10.881 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:38:10.881 INFO:teuthology.orchestra.run.smithi078.stdout:Total 4.4 MB/s | 173 kB 00:00 2023-12-21T03:38:10.881 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-12-21T03:38:10.889 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.2 MB/s | 173 kB 00:00 2023-12-21T03:38:10.889 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-21T03:38:10.890 INFO:teuthology.orchestra.run.smithi148.stdout:Total 3.2 MB/s | 173 kB 00:00 2023-12-21T03:38:10.890 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-21T03:38:10.891 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-12-21T03:38:10.892 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-12-21T03:38:10.901 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-21T03:38:10.901 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-21T03:38:10.940 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-12-21T03:38:10.944 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-12-21T03:38:10.950 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-21T03:38:10.953 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-21T03:38:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-12-21T03:38:11.067 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-21T03:38:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:11.331 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:11.551 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:11.661 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:12.240 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:12.240 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:12.240 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-12-21T03:38:12.240 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-21T03:38:12.240 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:38:12.240 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-12-21T03:38:12.328 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-21T03:38:12.328 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:12.328 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-21T03:38:12.328 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-21T03:38:12.328 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:38:12.328 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-21T03:38:12.485 DEBUG:teuthology.parallel:result is None 2023-12-21T03:38:12.525 DEBUG:teuthology.parallel:result is None 2023-12-21T03:38:12.525 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-21T03:38:12.525 INFO:teuthology.packaging:ref: None 2023-12-21T03:38:12.525 INFO:teuthology.packaging:tag: v16.2.4 2023-12-21T03:38:12.525 INFO:teuthology.packaging:branch: None 2023-12-21T03:38:12.526 INFO:teuthology.packaging:sha1: 639f0137318def3e5f393094e5494880043a8f32 2023-12-21T03:38:12.727 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-21T03:38:12.727 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-21T03:38:12.945 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-21T03:38:12.968 INFO:teuthology.orchestra.run.smithi078.stdout:16.2.4-0.el8 2023-12-21T03:38:12.968 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-21T03:38:12.969 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-21T03:38:12.970 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-21T03:38:12.970 INFO:teuthology.packaging:ref: None 2023-12-21T03:38:12.971 INFO:teuthology.packaging:tag: v16.2.4 2023-12-21T03:38:12.971 INFO:teuthology.packaging:branch: None 2023-12-21T03:38:12.971 INFO:teuthology.packaging:sha1: 639f0137318def3e5f393094e5494880043a8f32 2023-12-21T03:38:13.170 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-21T03:38:13.171 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-21T03:38:13.314 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-21T03:38:13.341 INFO:teuthology.orchestra.run.smithi148.stdout:16.2.4-0.el8 2023-12-21T03:38:13.342 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-21T03:38:13.342 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-21T03:38:13.344 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-21T03:38:13.345 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:38:13.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-21T03:38:13.374 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:38:13.374 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-21T03:38:13.412 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-21T03:38:13.412 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:38:13.412 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2023-12-21T03:38:13.442 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-21T03:38:13.509 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:38:13.509 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/daemon-helper 2023-12-21T03:38:13.537 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-21T03:38:13.606 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-21T03:38:13.606 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:38:13.606 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-21T03:38:13.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-21T03:38:13.699 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:38:13.699 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-21T03:38:13.728 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-21T03:38:13.796 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-21T03:38:13.796 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:38:13.796 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2023-12-21T03:38:13.822 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-21T03:38:13.890 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:38:13.891 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/stdin-killer 2023-12-21T03:38:13.918 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-21T03:38:13.985 INFO:teuthology.run_tasks:Running task print... 2023-12-21T03:38:13.995 INFO:teuthology.task.print:**** done install task... 2023-12-21T03:38:13.995 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-21T03:38:14.087 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '639f0137318def3e5f393094e5494880043a8f32'} 2023-12-21T03:38:14.087 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-12-21T03:38:14.087 INFO:tasks.cephadm:Cluster fsid is 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:14.087 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-21T03:38:14.088 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-12-21T03:38:14.088 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi078': '172.21.15.78', 'mon.smithi148': '172.21.15.148'} 2023-12-21T03:38:14.088 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-21T03:38:14.088 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname $(hostname -s) 2023-12-21T03:38:14.115 DEBUG:teuthology.orchestra.run.smithi148:> sudo hostname $(hostname -s) 2023-12-21T03:38:14.144 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-12-21T03:38:14.144 DEBUG:teuthology.orchestra.run.smithi078:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-21T03:38:14.291 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 21 03:38 /home/ubuntu/cephtest/cephadm 2023-12-21T03:38:14.291 DEBUG:teuthology.orchestra.run.smithi148:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-21T03:38:14.573 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 21 03:38 /home/ubuntu/cephtest/cephadm 2023-12-21T03:38:14.574 DEBUG:teuthology.orchestra.run.smithi078:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-21T03:38:14.593 DEBUG:teuthology.orchestra.run.smithi148:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-21T03:38:14.622 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph 2023-12-21T03:38:14.661 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph 2023-12-21T03:38:14.688 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /etc/ceph 2023-12-21T03:38:14.729 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 777 /etc/ceph 2023-12-21T03:38:14.755 INFO:tasks.cephadm:Writing seed config... 2023-12-21T03:38:14.756 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-12-21T03:38:14.756 INFO:tasks.cephadm: override: [client] debug client = 20 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-12-21T03:38:14.757 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-12-21T03:38:14.758 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-12-21T03:38:14.759 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-12-21T03:38:14.760 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:38:14.760 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-21T03:38:14.787 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 5ef0788c-9fb2-11ee-95a5-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-12-21T03:38:14.787 DEBUG:teuthology.orchestra.run.smithi078:mon.smithi078> sudo journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078.service 2023-12-21T03:38:14.829 INFO:tasks.cephadm:Bootstrapping... 2023-12-21T03:38:14.830 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.78 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-21T03:38:14.855 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:-- Logs begin at Thu 2023-12-21 03:26:39 UTC. -- 2023-12-21T03:38:15.118 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-21T03:38:15.923 INFO:teuthology.orchestra.run.smithi078.stderr:/bin/podman: 4.6.1 2023-12-21T03:38:15.941 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying podman|docker is present... 2023-12-21T03:38:15.941 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-21T03:38:15.961 INFO:teuthology.orchestra.run.smithi078.stderr:/bin/podman: 4.6.1 2023-12-21T03:38:15.972 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying lvm2 is present... 2023-12-21T03:38:15.972 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying time synchronization is in place... 2023-12-21T03:38:15.972 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled chrony.service 2023-12-21T03:38:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-21T03:38:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active chrony.service 2023-12-21T03:38:15.986 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-12-21T03:38:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-21T03:38:15.993 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: enabled 2023-12-21T03:38:15.993 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active chronyd.service 2023-12-21T03:38:15.999 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: active 2023-12-21T03:38:16.000 INFO:teuthology.orchestra.run.smithi078.stderr:Unit chronyd.service is enabled and running 2023-12-21T03:38:16.000 INFO:teuthology.orchestra.run.smithi078.stderr:Repeating the final host check... 2023-12-21T03:38:16.000 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-21T03:38:16.021 INFO:teuthology.orchestra.run.smithi078.stderr:/bin/podman: 4.6.1 2023-12-21T03:38:16.030 INFO:teuthology.orchestra.run.smithi078.stderr:podman|docker (/bin/podman) is present 2023-12-21T03:38:16.030 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl is present 2023-12-21T03:38:16.030 INFO:teuthology.orchestra.run.smithi078.stderr:lvcreate is present 2023-12-21T03:38:16.030 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled chrony.service 2023-12-21T03:38:16.035 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-21T03:38:16.035 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active chrony.service 2023-12-21T03:38:16.041 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-12-21T03:38:16.041 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-21T03:38:16.047 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: enabled 2023-12-21T03:38:16.048 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active chronyd.service 2023-12-21T03:38:16.053 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: active 2023-12-21T03:38:16.053 INFO:teuthology.orchestra.run.smithi078.stderr:Unit chronyd.service is enabled and running 2023-12-21T03:38:16.053 INFO:teuthology.orchestra.run.smithi078.stderr:Host looks OK 2023-12-21T03:38:16.054 INFO:teuthology.orchestra.run.smithi078.stderr:Cluster fsid: 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:16.054 INFO:teuthology.orchestra.run.smithi078.stderr:Acquiring lock 139970846759400 on /run/cephadm/5ef0788c-9fb2-11ee-95a5-87774f69a715.lock 2023-12-21T03:38:16.054 INFO:teuthology.orchestra.run.smithi078.stderr:Lock 139970846759400 acquired on /run/cephadm/5ef0788c-9fb2-11ee-95a5-87774f69a715.lock 2023-12-21T03:38:16.055 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying IP 172.21.15.78 port 3300 ... 2023-12-21T03:38:16.055 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying IP 172.21.15.78 port 6789 ... 2023-12-21T03:38:16.055 INFO:teuthology.orchestra.run.smithi078.stderr:Base mon IP is 172.21.15.78, final addrv is [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2023-12-21T03:38:16.055 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /sbin/ip route ls 2023-12-21T03:38:16.057 INFO:teuthology.orchestra.run.smithi078.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.78 metric 100 2023-12-21T03:38:16.057 INFO:teuthology.orchestra.run.smithi078.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.78 metric 100 2023-12-21T03:38:16.058 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /sbin/ip -6 route ls 2023-12-21T03:38:16.060 INFO:teuthology.orchestra.run.smithi078.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-12-21T03:38:16.060 INFO:teuthology.orchestra.run.smithi078.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-12-21T03:38:16.060 INFO:teuthology.orchestra.run.smithi078.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2023-12-21T03:38:16.061 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /sbin/ip -6 addr ls 2023-12-21T03:38:16.063 INFO:teuthology.orchestra.run.smithi078.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-21T03:38:16.063 INFO:teuthology.orchestra.run.smithi078.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-12-21T03:38:16.063 INFO:teuthology.orchestra.run.smithi078.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-21T03:38:16.063 INFO:teuthology.orchestra.run.smithi078.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-12-21T03:38:16.063 INFO:teuthology.orchestra.run.smithi078.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2023-12-21T03:38:16.063 INFO:teuthology.orchestra.run.smithi078.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-21T03:38:16.064 INFO:teuthology.orchestra.run.smithi078.stderr:Mon IP 172.21.15.78 is in CIDR network 172.21.0.0/20 2023-12-21T03:38:16.064 INFO:teuthology.orchestra.run.smithi078.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-21T03:38:16.065 INFO:teuthology.orchestra.run.smithi078.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-12-21T03:38:16.066 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-12-21T03:38:16.171 INFO:teuthology.orchestra.run.smithi078.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-12-21T03:38:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:/bin/podman: Getting image source signatures 2023-12-21T03:38:22.634 INFO:teuthology.orchestra.run.smithi078.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-12-21T03:38:22.634 INFO:teuthology.orchestra.run.smithi078.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-12-21T03:38:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-21T03:38:34.867 INFO:teuthology.orchestra.run.smithi078.stderr:/bin/podman: Writing manifest to image destination 2023-12-21T03:38:35.164 INFO:teuthology.orchestra.run.smithi078.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-21T03:38:35.179 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-12-21T03:38:35.748 INFO:teuthology.orchestra.run.smithi078.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-21T03:38:36.116 INFO:teuthology.orchestra.run.smithi078.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-21T03:38:36.116 INFO:teuthology.orchestra.run.smithi078.stderr:Extracting ceph user uid/gid from container image... 2023-12-21T03:38:36.116 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-12-21T03:38:36.411 INFO:teuthology.orchestra.run.smithi078.stderr:stat: 167 167 2023-12-21T03:38:36.749 INFO:teuthology.orchestra.run.smithi078.stderr:Creating initial keys... 2023-12-21T03:38:36.749 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-21T03:38:37.011 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-authtool: AQA8s4NlZzQ0OxAAG1sshoduPU9v0Q2J8tEfQA== 2023-12-21T03:38:37.338 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-21T03:38:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-authtool: AQA9s4Nl908lJRAA7vOFPLrA8ae7wFBIFyuxZA== 2023-12-21T03:38:38.004 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-21T03:38:38.278 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-authtool: AQA+s4Nleb23DxAAF6Wu4r6xRtkq1xCtTJA1jQ== 2023-12-21T03:38:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:Creating initial monmap... 2023-12-21T03:38:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpxahydbvw:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 --addv smithi078 [v2:172.21.15.78:3300,v1:172.21.15.78:6789] /tmp/monmap 2023-12-21T03:38:38.895 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-21T03:38:38.895 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:38.895 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-21T03:38:39.239 INFO:teuthology.orchestra.run.smithi078.stderr:monmaptool for smithi078 [v2:172.21.15.78:3300,v1:172.21.15.78:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-21T03:38:39.239 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/monmaptool: set fsid to 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:39.239 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-21T03:38:39.239 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-12-21T03:38:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:Creating mon... 2023-12-21T03:38:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078:/var/lib/ceph/mon/ceph-smithi078:z -v /tmp/ceph-tmpxmsvbkqp:/tmp/keyring:z -v /tmp/ceph-tmpxahydbvw:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi078 --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-12-21T03:38:39.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.503+0000 7f54f4d13700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-21T03:38:39.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.503+0000 7f54f4d13700 1 imported monmap: 2023-12-21T03:38:39.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: epoch 0 2023-12-21T03:38:39.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:39.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: last_changed 2023-12-21T03:38:38.889362+0000 2023-12-21T03:38:39.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: created 2023-12-21T03:38:38.889362+0000 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.smithi078 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.503+0000 7f54f4d13700 0 /usr/bin/ceph-mon: set fsid to 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: RocksDB version: 6.8.1 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Compile date May 13 2021 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: DB SUMMARY 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi078/store.db dir, Total Num: 0, files: 2023-12-21T03:38:39.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi078/store.db: 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.error_if_exists: 0 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.create_if_missing: 1 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.paranoid_checks: 1 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.env: 0x56005b88e1c0 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.fs: Posix File System 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.info_log: 0x56005d152ec0 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.statistics: (nil) 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.use_fsync: 0 2023-12-21T03:38:39.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.max_log_file_size: 0 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.allow_fallocate: 1 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.use_direct_reads: 0 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.db_log_dir: 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi078/store.db 2023-12-21T03:38:39.551 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.max_subcompactions: 1 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.max_background_flushes: -1 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.write_buffer_manager: 0x56005d159530 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-21T03:38:39.552 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-21T03:38:39.553 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-21T03:38:39.553 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-21T03:38:39.553 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-21T03:38:39.553 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-21T03:38:39.553 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-21T03:38:39.553 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-21T03:38:39.553 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.unordered_write: 0 2023-12-21T03:38:39.553 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-21T03:38:39.553 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.row_cache: None 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.wal_filter: None 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.preserve_deletes: 0 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.two_write_queues: 0 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.atomic_flush: 0 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-21T03:38:39.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.log_readahead_size: 0 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.max_background_jobs: 2 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.max_background_compactions: -1 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-21T03:38:39.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.max_open_files: -1 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Compression algorithms supported: 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: kZSTD supported: 0 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: kXpressCompression supported: 0 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: kLZ4Compression supported: 1 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: kBZip2Compression supported: 0 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: kZlibCompression supported: 1 2023-12-21T03:38:39.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: kSnappyCompression supported: 1 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.504+0000 7f54f4d13700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi078/store.db/MANIFEST-000001 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.merge_operator: 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_filter: None 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-21T03:38:39.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56005d06a120) 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: index_type: 0 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: checksum: 1 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-12-21T03:38:39.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_cache: 0x56005d0a0d10 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-12-21T03:38:39.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: format_version: 2 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_align: 0 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compression: NoCompression 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-21T03:38:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.num_levels: 7 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-21T03:38:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-21T03:38:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-21T03:38:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.table_properties_collectors: 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.inplace_update_support: 0 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.bloom_locality: 0 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.max_successive_merges: 0 2023-12-21T03:38:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.ttl: 2592000 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi078/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.505+0000 7f54f4d13700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.506+0000 7f54f4d13700 4 rocksdb: DB pointer 0x56005d165800 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.506+0000 7f54ddbac700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.506+0000 7f54ddbac700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-12-21T03:38:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-21T03:38:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-21T03:38:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-21T03:38:39.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-21T03:38:39.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-21T03:38:39.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-21T03:38:39.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-21T03:38:39.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-12-21T03:38:39.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.507+0000 7f54f4d13700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-12-21T03:38:39.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.507+0000 7f54f4d13700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-12-21T03:38:39.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-12-21T03:38:39.507+0000 7f54f4d13700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi078 for mon.smithi078 2023-12-21T03:38:39.867 INFO:teuthology.orchestra.run.smithi078.stderr:create mon.smithi078 on 2023-12-21T03:38:39.867 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:39.872 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl disable ceph.target 2023-12-21T03:38:39.879 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-12-21T03:38:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl enable ceph.target 2023-12-21T03:38:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-12-21T03:38:40.049 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl start ceph.target 2023-12-21T03:38:40.056 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl enable ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715.target 2023-12-21T03:38:40.062 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715.target. 2023-12-21T03:38:40.062 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715.target. 2023-12-21T03:38:40.141 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715.target 2023-12-21T03:38:40.148 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl daemon-reload 2023-12-21T03:38:40.231 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl stop ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078 2023-12-21T03:38:40.262 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl reset-failed ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078 2023-12-21T03:38:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to reset failed state of unit ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078.service: Unit ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078.service not loaded. 2023-12-21T03:38:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl enable ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078 2023-12-21T03:38:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Created symlink /etc/systemd/system/ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715.target.wants/ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078.service → /etc/systemd/system/ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@.service. 2023-12-21T03:38:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078 2023-12-21T03:38:40.397 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:40 smithi078 systemd[1]: Starting Ceph mon.smithi078 for 5ef0788c-9fb2-11ee-95a5-87774f69a715... 2023-12-21T03:38:40.706 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-21T03:38:40.716 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: disabled 2023-12-21T03:38:40.717 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active firewalld.service 2023-12-21T03:38:40.724 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-12-21T03:38:40.726 INFO:teuthology.orchestra.run.smithi078.stderr:firewalld.service is not enabled 2023-12-21T03:38:40.726 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to enable service . firewalld.service is not available 2023-12-21T03:38:40.726 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-21T03:38:40.726 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mon to start... 2023-12-21T03:38:40.726 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mon... 2023-12-21T03:38:40.726 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078:/var/lib/ceph/mon/ceph-smithi078:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-12-21T03:38:41.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:40 smithi078 bash[111284]: ad284a61445ea7bf7b7cacd9cc68abd86f4114dca845c17ca55be265c9965578 2023-12-21T03:38:41.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:40 smithi078 systemd[1]: Started Ceph mon.smithi078 for 5ef0788c-9fb2-11ee-95a5-87774f69a715. 2023-12-21T03:38:41.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: cluster 2023-12-21T03:38:40.709028+0000 mon.smithi078 (mon.0) 0 : [INF] mkfs 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:41.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: cluster 2023-12-21T03:38:40.687680+0000 mon.smithi078 (mon.0) 1 : cluster [INF] mon.smithi078 is new leader, mons smithi078 in quorum (ranks 0) 2023-12-21T03:38:41.124 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.121+0000 7fe008b14700 1 Processor -- start 2023-12-21T03:38:41.124 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.121+0000 7fe008b14700 1 -- start start 2023-12-21T03:38:41.124 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.122+0000 7fe008b14700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe004100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:41.124 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.122+0000 7fe008b14700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0040fc770 con 0x7fe004070390 2023-12-21T03:38:41.124 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.122+0000 7fe00259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe004100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.122+0000 7fe00259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe004100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50418/0 (socket says 172.21.15.78:50418) 2023-12-21T03:38:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.122+0000 7fe00259c700 1 -- 172.21.15.78:0/2372814078 learned_addr learned my addr 172.21.15.78:0/2372814078 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.123+0000 7fe00259c700 1 -- 172.21.15.78:0/2372814078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe004101270 con 0x7fe004070390 2023-12-21T03:38:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.123+0000 7fe00259c700 1 --2- 172.21.15.78:0/2372814078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe004100d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fdfec00b5f0 tx=0x7fdfec006ee0).ready entity=mon.0 client_cookie=1b2e7cd92553c0e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.123+0000 7fe00159a700 1 -- 172.21.15.78:0/2372814078 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfec00d600 con 0x7fe004070390 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.123+0000 7fe00159a700 1 -- 172.21.15.78:0/2372814078 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec00d760 con 0x7fe004070390 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.123+0000 7fe00159a700 1 -- 172.21.15.78:0/2372814078 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfec00da30 con 0x7fe004070390 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.123+0000 7fe008b14700 1 -- 172.21.15.78:0/2372814078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 msgr2=0x7fe004100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.123+0000 7fe008b14700 1 --2- 172.21.15.78:0/2372814078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe004100d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fdfec00b5f0 tx=0x7fdfec006ee0).stop 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.123+0000 7fe008b14700 1 -- 172.21.15.78:0/2372814078 shutdown_connections 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.123+0000 7fe008b14700 1 --2- 172.21.15.78:0/2372814078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe004100d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.123+0000 7fe008b14700 1 -- 172.21.15.78:0/2372814078 >> 172.21.15.78:0/2372814078 conn(0x7fe0040f63c0 msgr2=0x7fe0040f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.124+0000 7fe008b14700 1 -- 172.21.15.78:0/2372814078 shutdown_connections 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.124+0000 7fe008b14700 1 -- 172.21.15.78:0/2372814078 wait complete. 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.124+0000 7fe008b14700 1 Processor -- start 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.124+0000 7fe008b14700 1 -- start start 2023-12-21T03:38:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.124+0000 7fe008b14700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe00410c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.124+0000 7fe008b14700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00410c9a0 con 0x7fe004070390 2023-12-21T03:38:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fe00259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe00410c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fe00259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe00410c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50426/0 (socket says 172.21.15.78:50426) 2023-12-21T03:38:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fe00259c700 1 -- 172.21.15.78:0/1394328363 learned_addr learned my addr 172.21.15.78:0/1394328363 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fe00259c700 1 -- 172.21.15.78:0/1394328363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfec00b200 con 0x7fe004070390 2023-12-21T03:38:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fe00259c700 1 --2- 172.21.15.78:0/1394328363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe00410c4b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdfec006c70 tx=0x7fdfec016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fdffaffd700 1 -- 172.21.15.78:0/1394328363 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfec011b40 con 0x7fe004070390 2023-12-21T03:38:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fdffaffd700 1 -- 172.21.15.78:0/1394328363 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec011ca0 con 0x7fe004070390 2023-12-21T03:38:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fe008b14700 1 -- 172.21.15.78:0/1394328363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00410cba0 con 0x7fe004070390 2023-12-21T03:38:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fdffaffd700 1 -- 172.21.15.78:0/1394328363 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfec01d430 con 0x7fe004070390 2023-12-21T03:38:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fe008b14700 1 -- 172.21.15.78:0/1394328363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00410cfb0 con 0x7fe004070390 2023-12-21T03:38:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fdffaffd700 1 -- 172.21.15.78:0/1394328363 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fdfec00f070 con 0x7fe004070390 2023-12-21T03:38:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.125+0000 7fdffaffd700 1 -- 172.21.15.78:0/1394328363 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fdfec027460 con 0x7fe004070390 2023-12-21T03:38:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.126+0000 7fe008b14700 1 -- 172.21.15.78:0/1394328363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe004045ad0 con 0x7fe004070390 2023-12-21T03:38:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.128+0000 7fdffaffd700 1 -- 172.21.15.78:0/1394328363 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fdfec016070 con 0x7fe004070390 2023-12-21T03:38:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.195+0000 7fe008b14700 1 -- 172.21.15.78:0/1394328363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe004106050 con 0x7fe004070390 2023-12-21T03:38:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.195+0000 7fdffaffd700 1 -- 172.21.15.78:0/1394328363 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fdfec036070 con 0x7fe004070390 2023-12-21T03:38:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: cluster: 2023-12-21T03:38:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: id: 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-12-21T03:38:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: services: 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi078 (age 0.489276s) 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mgr: no daemons active 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: data: 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: pgs: 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.196+0000 7fe008b14700 1 -- 172.21.15.78:0/1394328363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 msgr2=0x7fe00410c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.196+0000 7fe008b14700 1 --2- 172.21.15.78:0/1394328363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe00410c4b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdfec006c70 tx=0x7fdfec016000).stop 2023-12-21T03:38:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.196+0000 7fe008b14700 1 -- 172.21.15.78:0/1394328363 shutdown_connections 2023-12-21T03:38:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.196+0000 7fe008b14700 1 --2- 172.21.15.78:0/1394328363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004070390 0x7fe00410c4b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.196+0000 7fe008b14700 1 -- 172.21.15.78:0/1394328363 >> 172.21.15.78:0/1394328363 conn(0x7fe0040f63c0 msgr2=0x7fe0040f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.196+0000 7fe008b14700 1 -- 172.21.15.78:0/1394328363 shutdown_connections 2023-12-21T03:38:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:41.196+0000 7fe008b14700 1 -- 172.21.15.78:0/1394328363 wait complete. 2023-12-21T03:38:41.563 INFO:teuthology.orchestra.run.smithi078.stderr:mon is available 2023-12-21T03:38:41.564 INFO:teuthology.orchestra.run.smithi078.stderr:Assimilating anything we can from ceph.conf... 2023-12-21T03:38:41.564 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z -v /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078:/var/lib/ceph/mon/ceph-smithi078:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi078/config 2023-12-21T03:38:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: cluster 2023-12-21T03:38:40.707632+0000 mon.smithi078 (mon.0) 2 : cluster [INF] mon.smithi078 is new leader, mons smithi078 in quorum (ranks 0) 2023-12-21T03:38:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: cluster 2023-12-21T03:38:40.707909+0000 mon.smithi078 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi078=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} 2023-12-21T03:38:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: cluster 2023-12-21T03:38:40.711212+0000 mon.smithi078 (mon.0) 4 : cluster [DBG] fsmap 2023-12-21T03:38:41.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: cluster 2023-12-21T03:38:40.714666+0000 mon.smithi078 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-21T03:38:41.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: cluster 2023-12-21T03:38:40.715050+0000 mon.smithi078 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-12-21T03:38:41.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: audit 2023-12-21T03:38:41.196873+0000 mon.smithi078 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.78:0/1394328363' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-21T03:38:42.244 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.240+0000 7fb5c94d7700 1 Processor -- start 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.240+0000 7fb5c94d7700 1 -- start start 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.241+0000 7fb5c94d7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c4100d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.241+0000 7fb5c94d7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c40fc760 con 0x7fb5c4070420 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.241+0000 7fb5c2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c4100d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.241+0000 7fb5c2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c4100d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50442/0 (socket says 172.21.15.78:50442) 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.241+0000 7fb5c2ffd700 1 -- 172.21.15.78:0/2272752203 learned_addr learned my addr 172.21.15.78:0/2272752203 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.241+0000 7fb5c2ffd700 1 -- 172.21.15.78:0/2272752203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c4101260 con 0x7fb5c4070420 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.241+0000 7fb5c2ffd700 1 --2- 172.21.15.78:0/2272752203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c4100d70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb5b800b5f0 tx=0x7fb5b8006ee0).ready entity=mon.0 client_cookie=d592f52feff7656 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.242+0000 7fb5c1ffb700 1 -- 172.21.15.78:0/2272752203 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5b800d600 con 0x7fb5c4070420 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.242+0000 7fb5c1ffb700 1 -- 172.21.15.78:0/2272752203 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b800d760 con 0x7fb5c4070420 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.242+0000 7fb5c1ffb700 1 -- 172.21.15.78:0/2272752203 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5b800da30 con 0x7fb5c4070420 2023-12-21T03:38:42.245 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.242+0000 7fb5c94d7700 1 -- 172.21.15.78:0/2272752203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 msgr2=0x7fb5c4100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:42.246 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.242+0000 7fb5c94d7700 1 --2- 172.21.15.78:0/2272752203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c4100d70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb5b800b5f0 tx=0x7fb5b8006ee0).stop 2023-12-21T03:38:42.246 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.242+0000 7fb5c94d7700 1 -- 172.21.15.78:0/2272752203 shutdown_connections 2023-12-21T03:38:42.246 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.242+0000 7fb5c94d7700 1 --2- 172.21.15.78:0/2272752203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c4100d70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:42.246 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.242+0000 7fb5c94d7700 1 -- 172.21.15.78:0/2272752203 >> 172.21.15.78:0/2272752203 conn(0x7fb5c40f63b0 msgr2=0x7fb5c40f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:42.246 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.242+0000 7fb5c94d7700 1 -- 172.21.15.78:0/2272752203 shutdown_connections 2023-12-21T03:38:42.246 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.242+0000 7fb5c94d7700 1 -- 172.21.15.78:0/2272752203 wait complete. 2023-12-21T03:38:42.246 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.243+0000 7fb5c94d7700 1 Processor -- start 2023-12-21T03:38:42.246 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.243+0000 7fb5c94d7700 1 -- start start 2023-12-21T03:38:42.246 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.243+0000 7fb5c94d7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c410c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:42.246 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.243+0000 7fb5c94d7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c410c980 con 0x7fb5c4070420 2023-12-21T03:38:42.248 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.243+0000 7fb5c2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c410c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:42.248 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.243+0000 7fb5c2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c410c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50446/0 (socket says 172.21.15.78:50446) 2023-12-21T03:38:42.248 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.243+0000 7fb5c2ffd700 1 -- 172.21.15.78:0/656331178 learned_addr learned my addr 172.21.15.78:0/656331178 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:42.248 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.243+0000 7fb5c2ffd700 1 -- 172.21.15.78:0/656331178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b800b200 con 0x7fb5c4070420 2023-12-21T03:38:42.248 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.244+0000 7fb5c2ffd700 1 --2- 172.21.15.78:0/656331178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c410c490 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fb5b8006c70 tx=0x7fb5b8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:42.248 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.244+0000 7fb5ab7fe700 1 -- 172.21.15.78:0/656331178 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5b8011b40 con 0x7fb5c4070420 2023-12-21T03:38:42.248 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.244+0000 7fb5ab7fe700 1 -- 172.21.15.78:0/656331178 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b8011ca0 con 0x7fb5c4070420 2023-12-21T03:38:42.248 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.244+0000 7fb5c94d7700 1 -- 172.21.15.78:0/656331178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c410cb80 con 0x7fb5c4070420 2023-12-21T03:38:42.249 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.244+0000 7fb5ab7fe700 1 -- 172.21.15.78:0/656331178 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5b801d430 con 0x7fb5c4070420 2023-12-21T03:38:42.249 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.244+0000 7fb5c94d7700 1 -- 172.21.15.78:0/656331178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c410cf90 con 0x7fb5c4070420 2023-12-21T03:38:42.249 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.244+0000 7fb5ab7fe700 1 -- 172.21.15.78:0/656331178 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb5b800f070 con 0x7fb5c4070420 2023-12-21T03:38:42.249 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.244+0000 7fb5ab7fe700 1 -- 172.21.15.78:0/656331178 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb5b8027460 con 0x7fb5c4070420 2023-12-21T03:38:42.249 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.245+0000 7fb5c94d7700 1 -- 172.21.15.78:0/656331178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c4045ad0 con 0x7fb5c4070420 2023-12-21T03:38:42.249 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.247+0000 7fb5ab7fe700 1 -- 172.21.15.78:0/656331178 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb5b8016070 con 0x7fb5c4070420 2023-12-21T03:38:42.312 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.310+0000 7fb5c94d7700 1 -- 172.21.15.78:0/656331178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb5c41013f0 con 0x7fb5c4070420 2023-12-21T03:38:42.315 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.313+0000 7fb5ab7fe700 1 -- 172.21.15.78:0/656331178 <== mon.0 v2:172.21.15.78:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb5b801d5e0 con 0x7fb5c4070420 2023-12-21T03:38:42.315 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.313+0000 7fb5ab7fe700 1 -- 172.21.15.78:0/656331178 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7fb5b8036020 con 0x7fb5c4070420 2023-12-21T03:38:42.318 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.318 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: [global] 2023-12-21T03:38:42.318 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: fsid = 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:42.318 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2023-12-21T03:38:42.318 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-12-21T03:38:42.318 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-12-21T03:38:42.318 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-12-21T03:38:42.318 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-12-21T03:38:42.318 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.318 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: [mgr] 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: [osd] 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.315+0000 7fb5c94d7700 1 -- 172.21.15.78:0/656331178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 msgr2=0x7fb5c410c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.315+0000 7fb5c94d7700 1 --2- 172.21.15.78:0/656331178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c410c490 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fb5b8006c70 tx=0x7fb5b8016000).stop 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.315+0000 7fb5c94d7700 1 -- 172.21.15.78:0/656331178 shutdown_connections 2023-12-21T03:38:42.319 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.315+0000 7fb5c94d7700 1 --2- 172.21.15.78:0/656331178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c4070420 0x7fb5c410c490 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.315+0000 7fb5c94d7700 1 -- 172.21.15.78:0/656331178 >> 172.21.15.78:0/656331178 conn(0x7fb5c40f63b0 msgr2=0x7fb5c40f8110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.315+0000 7fb5c94d7700 1 -- 172.21.15.78:0/656331178 shutdown_connections 2023-12-21T03:38:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:42.315+0000 7fb5c94d7700 1 -- 172.21.15.78:0/656331178 wait complete. 2023-12-21T03:38:42.655 INFO:teuthology.orchestra.run.smithi078.stderr:Generating new minimal ceph.conf... 2023-12-21T03:38:42.655 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z -v /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078:/var/lib/ceph/mon/ceph-smithi078:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi078/config 2023-12-21T03:38:43.012 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.010+0000 7f9214e85700 1 Processor -- start 2023-12-21T03:38:43.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.010+0000 7f9214e85700 1 -- start start 2023-12-21T03:38:43.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.010+0000 7f9214e85700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f9210100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:43.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.010+0000 7f9214e85700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92100fc770 con 0x7f9210070390 2023-12-21T03:38:43.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.010+0000 7f920e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f9210100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:43.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.010+0000 7f920e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f9210100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50452/0 (socket says 172.21.15.78:50452) 2023-12-21T03:38:43.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.010+0000 7f920e59c700 1 -- 172.21.15.78:0/233324260 learned_addr learned my addr 172.21.15.78:0/233324260 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:43.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.011+0000 7f920e59c700 1 -- 172.21.15.78:0/233324260 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9210101270 con 0x7f9210070390 2023-12-21T03:38:43.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.011+0000 7f920e59c700 1 --2- 172.21.15.78:0/233324260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f9210100d80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f920400b5f0 tx=0x7f9204006ee0).ready entity=mon.0 client_cookie=b588ec250a6df918 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:43.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.011+0000 7f920d59a700 1 -- 172.21.15.78:0/233324260 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f920400d600 con 0x7f9210070390 2023-12-21T03:38:43.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.011+0000 7f920d59a700 1 -- 172.21.15.78:0/233324260 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f920400dbd0 con 0x7f9210070390 2023-12-21T03:38:43.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.011+0000 7f920d59a700 1 -- 172.21.15.78:0/233324260 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9204011c00 con 0x7f9210070390 2023-12-21T03:38:43.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.011+0000 7f9214e85700 1 -- 172.21.15.78:0/233324260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 msgr2=0x7f9210100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:43.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.011+0000 7f9214e85700 1 --2- 172.21.15.78:0/233324260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f9210100d80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f920400b5f0 tx=0x7f9204006ee0).stop 2023-12-21T03:38:43.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.012+0000 7f9214e85700 1 -- 172.21.15.78:0/233324260 shutdown_connections 2023-12-21T03:38:43.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.012+0000 7f9214e85700 1 --2- 172.21.15.78:0/233324260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f9210100d80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:43.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.012+0000 7f9214e85700 1 -- 172.21.15.78:0/233324260 >> 172.21.15.78:0/233324260 conn(0x7f92100f63c0 msgr2=0x7f92100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:43.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.012+0000 7f9214e85700 1 -- 172.21.15.78:0/233324260 shutdown_connections 2023-12-21T03:38:43.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.012+0000 7f9214e85700 1 -- 172.21.15.78:0/233324260 wait complete. 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.012+0000 7f9214e85700 1 Processor -- start 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.012+0000 7f9214e85700 1 -- start start 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.012+0000 7f9214e85700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f921010ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.012+0000 7f9214e85700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921010d1b0 con 0x7f9210070390 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.012+0000 7f920e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f921010ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.013+0000 7f920e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f921010ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50458/0 (socket says 172.21.15.78:50458) 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.013+0000 7f920e59c700 1 -- 172.21.15.78:0/2929125714 learned_addr learned my addr 172.21.15.78:0/2929125714 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.013+0000 7f920e59c700 1 -- 172.21.15.78:0/2929125714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920400b200 con 0x7f9210070390 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.013+0000 7f920e59c700 1 --2- 172.21.15.78:0/2929125714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f921010ccc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f920400b190 tx=0x7f9204006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.013+0000 7f91feffd700 1 -- 172.21.15.78:0/2929125714 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f920401b410 con 0x7f9210070390 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.013+0000 7f9214e85700 1 -- 172.21.15.78:0/2929125714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921010d3b0 con 0x7f9210070390 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.013+0000 7f91feffd700 1 -- 172.21.15.78:0/2929125714 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f920401b9e0 con 0x7f9210070390 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.013+0000 7f91feffd700 1 -- 172.21.15.78:0/2929125714 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f920401a880 con 0x7f9210070390 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.013+0000 7f9214e85700 1 -- 172.21.15.78:0/2929125714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921010d7c0 con 0x7f9210070390 2023-12-21T03:38:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.013+0000 7f91feffd700 1 -- 172.21.15.78:0/2929125714 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f920400f070 con 0x7f9210070390 2023-12-21T03:38:43.019 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.013+0000 7f91feffd700 1 -- 172.21.15.78:0/2929125714 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9204025c90 con 0x7f9210070390 2023-12-21T03:38:43.019 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.015+0000 7f9214e85700 1 -- 172.21.15.78:0/2929125714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9210045ad0 con 0x7f9210070390 2023-12-21T03:38:43.019 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.017+0000 7f91feffd700 1 -- 172.21.15.78:0/2929125714 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9204016070 con 0x7f9210070390 2023-12-21T03:38:43.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: audit 2023-12-21T03:38:42.311689+0000 mon.smithi078 (mon.0) 8 : audit [INF] from='client.? 172.21.15.78:0/656331178' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-21T03:38:43.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: audit 2023-12-21T03:38:42.314758+0000 mon.smithi078 (mon.0) 9 : audit [INF] from='client.? 172.21.15.78:0/656331178' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-21T03:38:43.083 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.081+0000 7f9214e85700 1 -- 172.21.15.78:0/2929125714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9210101d70 con 0x7f9210070390 2023-12-21T03:38:43.083 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.081+0000 7f91feffd700 1 -- 172.21.15.78:0/2929125714 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f920401bb50 con 0x7f9210070390 2023-12-21T03:38:43.084 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.082+0000 7f9214e85700 1 -- 172.21.15.78:0/2929125714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 msgr2=0x7f921010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:43.084 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.082+0000 7f9214e85700 1 --2- 172.21.15.78:0/2929125714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f921010ccc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f920400b190 tx=0x7f9204006ee0).stop 2023-12-21T03:38:43.084 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.082+0000 7f9214e85700 1 -- 172.21.15.78:0/2929125714 shutdown_connections 2023-12-21T03:38:43.085 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.082+0000 7f9214e85700 1 --2- 172.21.15.78:0/2929125714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9210070390 0x7f921010ccc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:43.085 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.082+0000 7f9214e85700 1 -- 172.21.15.78:0/2929125714 >> 172.21.15.78:0/2929125714 conn(0x7f92100f63c0 msgr2=0x7f92100f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:43.085 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.082+0000 7f9214e85700 1 -- 172.21.15.78:0/2929125714 shutdown_connections 2023-12-21T03:38:43.085 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:43.082+0000 7f9214e85700 1 -- 172.21.15.78:0/2929125714 wait complete. 2023-12-21T03:38:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:Restarting the monitor... 2023-12-21T03:38:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078 2023-12-21T03:38:43.720 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:43 smithi078 systemd[1]: Stopping Ceph mon.smithi078 for 5ef0788c-9fb2-11ee-95a5-87774f69a715... 2023-12-21T03:38:43.720 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: debug 2023-12-21T03:38:43.542+0000 7f026dac3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi078 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-21T03:38:43.720 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111299]: debug 2023-12-21T03:38:43.542+0000 7f026dac3700 -1 mon.smithi078@0(leader) e1 *** Got Signal Terminated *** 2023-12-21T03:38:44.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:43 smithi078 podman[111620]: ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078 2023-12-21T03:38:44.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:43 smithi078 systemd[1]: ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078.service: Succeeded. 2023-12-21T03:38:44.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:43 smithi078 systemd[1]: Stopped Ceph mon.smithi078 for 5ef0788c-9fb2-11ee-95a5-87774f69a715. 2023-12-21T03:38:44.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:43 smithi078 systemd[1]: Starting Ceph mon.smithi078 for 5ef0788c-9fb2-11ee-95a5-87774f69a715... 2023-12-21T03:38:44.188 INFO:teuthology.orchestra.run.smithi078.stderr:Setting mon public_network to 172.21.0.0/20 2023-12-21T03:38:44.188 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-12-21T03:38:44.545 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.541+0000 7f9c9083b700 1 Processor -- start 2023-12-21T03:38:44.545 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.541+0000 7f9c9083b700 1 -- start start 2023-12-21T03:38:44.545 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.542+0000 7f9c9083b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:44.545 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.542+0000 7f9c9083b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8c0728b0 con 0x7f9c8c072490 2023-12-21T03:38:44.545 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.542+0000 7f9c8ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:44.545 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.542+0000 7f9c8ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50460/0 (socket says 172.21.15.78:50460) 2023-12-21T03:38:44.547 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.542+0000 7f9c8ad9d700 1 -- 172.21.15.78:0/2985335001 learned_addr learned my addr 172.21.15.78:0/2985335001 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:44.547 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.542+0000 7f9c8ad9d700 1 -- 172.21.15.78:0/2985335001 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8c070f20 con 0x7f9c8c072490 2023-12-21T03:38:44.547 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.543+0000 7f9c8ad9d700 1 --2- 172.21.15.78:0/2985335001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c070a30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9c7c00b5f0 tx=0x7f9c7c006ee0).ready entity=mon.0 client_cookie=63a0074c2fffae67 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:44.547 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.543+0000 7f9c89d9b700 1 -- 172.21.15.78:0/2985335001 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c7c00d600 con 0x7f9c8c072490 2023-12-21T03:38:44.547 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.543+0000 7f9c89d9b700 1 -- 172.21.15.78:0/2985335001 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9c7c00dbd0 con 0x7f9c8c072490 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.543+0000 7f9c89d9b700 1 -- 172.21.15.78:0/2985335001 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c7c011c00 con 0x7f9c8c072490 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.543+0000 7f9c9083b700 1 -- 172.21.15.78:0/2985335001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 msgr2=0x7f9c8c070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.543+0000 7f9c9083b700 1 --2- 172.21.15.78:0/2985335001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c070a30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9c7c00b5f0 tx=0x7f9c7c006ee0).stop 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.543+0000 7f9c9083b700 1 -- 172.21.15.78:0/2985335001 shutdown_connections 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.543+0000 7f9c9083b700 1 --2- 172.21.15.78:0/2985335001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c070a30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.543+0000 7f9c9083b700 1 -- 172.21.15.78:0/2985335001 >> 172.21.15.78:0/2985335001 conn(0x7f9c8c0f7380 msgr2=0x7f9c8c0f97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.543+0000 7f9c9083b700 1 -- 172.21.15.78:0/2985335001 shutdown_connections 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.543+0000 7f9c9083b700 1 -- 172.21.15.78:0/2985335001 wait complete. 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.544+0000 7f9c9083b700 1 Processor -- start 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.544+0000 7f9c9083b700 1 -- start start 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.544+0000 7f9c9083b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c10cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.544+0000 7f9c9083b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8c10cff0 con 0x7f9c8c072490 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.544+0000 7f9c8ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c10cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.544+0000 7f9c8ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c10cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50466/0 (socket says 172.21.15.78:50466) 2023-12-21T03:38:44.548 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.544+0000 7f9c8ad9d700 1 -- 172.21.15.78:0/2102507395 learned_addr learned my addr 172.21.15.78:0/2102507395 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:44.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.545+0000 7f9c8ad9d700 1 -- 172.21.15.78:0/2102507395 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c00b200 con 0x7f9c8c072490 2023-12-21T03:38:44.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.545+0000 7f9c8ad9d700 1 --2- 172.21.15.78:0/2102507395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c10cb00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f9c7c00b190 tx=0x7f9c7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:44.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.545+0000 7f9c837fe700 1 -- 172.21.15.78:0/2102507395 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c7c011a20 con 0x7f9c8c072490 2023-12-21T03:38:44.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.545+0000 7f9c9083b700 1 -- 172.21.15.78:0/2102507395 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8c10d1f0 con 0x7f9c8c072490 2023-12-21T03:38:44.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.545+0000 7f9c837fe700 1 -- 172.21.15.78:0/2102507395 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9c7c01b880 con 0x7f9c8c072490 2023-12-21T03:38:44.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.545+0000 7f9c837fe700 1 -- 172.21.15.78:0/2102507395 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c7c01a710 con 0x7f9c8c072490 2023-12-21T03:38:44.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.545+0000 7f9c9083b700 1 -- 172.21.15.78:0/2102507395 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8c10d600 con 0x7f9c8c072490 2023-12-21T03:38:44.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.545+0000 7f9c837fe700 1 -- 172.21.15.78:0/2102507395 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f9c7c00f070 con 0x7f9c8c072490 2023-12-21T03:38:44.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.545+0000 7f9c837fe700 1 -- 172.21.15.78:0/2102507395 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9c7c025b10 con 0x7f9c8c072490 2023-12-21T03:38:44.549 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.546+0000 7f9c9083b700 1 -- 172.21.15.78:0/2102507395 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c8c045ad0 con 0x7f9c8c072490 2023-12-21T03:38:44.550 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.548+0000 7f9c837fe700 1 -- 172.21.15.78:0/2102507395 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9c7c016070 con 0x7f9c8c072490 2023-12-21T03:38:44.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.178+0000 7f630dfa0700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-21T03:38:44.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.178+0000 7f630dfa0700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-12-21T03:38:44.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.178+0000 7f630dfa0700 0 pidfile_write: ignore empty --pid-file 2023-12-21T03:38:44.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.182+0000 7f630dfa0700 0 load: jerasure load: lrc load: isa 2023-12-21T03:38:44.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: RocksDB version: 6.8.1 2023-12-21T03:38:44.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Compile date May 13 2021 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: DB SUMMARY 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: CURRENT file: CURRENT 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: IDENTITY file: IDENTITY 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi078/store.db dir, Total Num: 1, files: 000004.sst 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 bash[111691]: cfa63aa627cc61e670c51547b3fc8163fa6cbc0800454b511a0827dc28365b59 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi078/store.db: 000006.log size: 92912 ; 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.error_if_exists: 0 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.create_if_missing: 0 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.paranoid_checks: 1 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.env: 0x55f86f2091c0 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.fs: Posix File System 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.info_log: 0x55f8717c7960 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.statistics: (nil) 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.use_fsync: 0 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_log_file_size: 0 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-21T03:38:44.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-21T03:38:44.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-21T03:38:44.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.allow_fallocate: 1 2023-12-21T03:38:44.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-21T03:38:44.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-21T03:38:44.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.use_direct_reads: 0 2023-12-21T03:38:44.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-21T03:38:44.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-21T03:38:44.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.db_log_dir: 2023-12-21T03:38:44.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi078/store.db 2023-12-21T03:38:44.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-21T03:38:44.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_subcompactions: 1 2023-12-21T03:38:44.571 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_background_flushes: -1 2023-12-21T03:38:44.571 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-21T03:38:44.571 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-21T03:38:44.571 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-21T03:38:44.571 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-21T03:38:44.571 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-21T03:38:44.571 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-21T03:38:44.571 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-21T03:38:44.571 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.write_buffer_manager: 0x55f87243c030 2023-12-21T03:38:44.571 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.unordered_write: 0 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.row_cache: None 2023-12-21T03:38:44.572 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.wal_filter: None 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.preserve_deletes: 0 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.two_write_queues: 0 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.atomic_flush: 0 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.log_readahead_size: 0 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_background_jobs: 2 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_background_compactions: -1 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-21T03:38:44.573 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_open_files: -1 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Compression algorithms supported: 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: kZSTD supported: 0 2023-12-21T03:38:44.574 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: kXpressCompression supported: 0 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: kLZ4Compression supported: 1 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: kBZip2Compression supported: 0 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: kZlibCompression supported: 1 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: kSnappyCompression supported: 1 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi078/store.db/MANIFEST-000005 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.merge_operator: 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_filter: None 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-21T03:38:44.575 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f8716de128) 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cache_index_and_filter_blocks: 1 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: pin_top_level_index_and_filter: 1 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: index_type: 0 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: data_block_index_type: 0 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: index_shortening: 1 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: data_block_hash_table_util_ratio: 0.750000 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: hash_index_allow_collision: 1 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: checksum: 1 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: no_block_cache: 0 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: block_cache: 0x55f871714f10 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: block_cache_name: BinnedLRUCache 2023-12-21T03:38:44.576 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: block_cache_options: 2023-12-21T03:38:44.577 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: capacity : 536870912 2023-12-21T03:38:44.577 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: num_shard_bits : 4 2023-12-21T03:38:44.578 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: strict_capacity_limit : 0 2023-12-21T03:38:44.578 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: high_pri_pool_ratio: 0.000 2023-12-21T03:38:44.578 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: block_cache_compressed: (nil) 2023-12-21T03:38:44.578 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: persistent_cache: (nil) 2023-12-21T03:38:44.578 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: block_size: 4096 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: block_size_deviation: 10 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: block_restart_interval: 16 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: index_block_restart_interval: 1 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: metadata_block_size: 4096 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: partition_filters: 0 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: use_delta_encoding: 1 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: whole_key_filtering: 1 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: verify_compression: 0 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: read_amp_bytes_per_bit: 0 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: format_version: 2 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: enable_index_compression: 1 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: block_align: 0 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.579 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compression: NoCompression 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.num_levels: 7 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-21T03:38:44.580 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-21T03:38:44.581 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-21T03:38:44.582 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.table_properties_collectors: 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.inplace_update_support: 0 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-21T03:38:44.583 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.bloom_locality: 0 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.max_successive_merges: 0 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.ttl: 2592000 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.183+0000 7f630dfa0700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.184+0000 7f630dfa0700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi078/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.184+0000 7f630dfa0700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.184+0000 7f630dfa0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703129924185296, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.184+0000 7f630dfa0700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-12-21T03:38:44.584 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.184+0000 7f630dfa0700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.185+0000 7f630dfa0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703129924186133, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88692, "table_properties": {"data_size": 86807, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79948, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1703129924, "oldest_key_time": 3, "file_creation_time": 0}} 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.185+0000 7f630dfa0700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.185+0000 7f630dfa0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703129924186519, "job": 1, "event": "recovery_finished"} 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 systemd[1]: Started Ceph mon.smithi078 for 5ef0788c-9fb2-11ee-95a5-87774f69a715. 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.186+0000 7f630dfa0700 4 rocksdb: DB pointer 0x55f8717d9800 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.186+0000 7f62f4db8700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.186+0000 7f62f4db8700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ** DB Stats ** 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-21T03:38:44.585 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ** Compaction Stats [default] ** 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.4 0.00 0.00 1 0.001 0 0 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.4 0.00 0.00 1 0.001 0 0 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.4 0.00 0.00 1 0.001 0 0 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ** Compaction Stats [default] ** 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.4 0.00 0.00 1 0.001 0 0 2023-12-21T03:38:44.586 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: AddFile(Total Files): cumulative 0, interval 0 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: AddFile(Keys): cumulative 0, interval 0 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Cumulative compaction: 0.00 GB write, 29.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Interval compaction: 0.00 GB write, 29.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ** File Read Latency Histogram By Level [default] ** 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.587 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ** Compaction Stats [default] ** 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.4 0.00 0.00 1 0.001 0 0 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.4 0.00 0.00 1 0.001 0 0 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ** Compaction Stats [default] ** 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.4 0.00 0.00 1 0.001 0 0 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-21T03:38:44.588 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: AddFile(Total Files): cumulative 0, interval 0 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: AddFile(Keys): cumulative 0, interval 0 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Cumulative compaction: 0.00 GB write, 29.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ** File Read Latency Histogram By Level [default] ** 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.186+0000 7f630dfa0700 0 starting mon.smithi078 rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi078 fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.187+0000 7f630dfa0700 1 mon.smithi078@-1(???) e1 preinit fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.188+0000 7f630dfa0700 0 mon.smithi078@-1(???).mds e1 new map 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.188+0000 7f630dfa0700 0 mon.smithi078@-1(???).mds e1 print_map 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: e1 2023-12-21T03:38:44.589 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: legacy client fscid: -1 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: No filesystems configured 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.188+0000 7f630dfa0700 0 mon.smithi078@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.188+0000 7f630dfa0700 0 mon.smithi078@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.188+0000 7f630dfa0700 0 mon.smithi078@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.188+0000 7f630dfa0700 0 mon.smithi078@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:42.311689+0000 mon.smithi078 (mon.0) 8 : audit [INF] from='client.? 172.21.15.78:0/656331178' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:42.314758+0000 mon.smithi078 (mon.0) 9 : audit [INF] from='client.? 172.21.15.78:0/656331178' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.188+0000 7f630dfa0700 1 mon.smithi078@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta expand map: {default=false} 2023-12-21T03:38:44.590 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta from 'false' to 'false' 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta expanded map: {default=false} 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta expand map: {default=info} 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta from 'info' to 'info' 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta expanded map: {default=info} 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta expand map: {default=daemon} 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta expanded map: {default=daemon} 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta expand map: {default=debug} 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta from 'debug' to 'debug' 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:38:44.189+0000 7f630dfa0700 20 expand_channel_meta expanded map: {default=debug} 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:44.204783+0000 mon.smithi078 (mon.0) 1 : cluster [INF] mon.smithi078 is new leader, mons smithi078 in quorum (ranks 0) 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:44.204853+0000 mon.smithi078 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi078=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:44.204892+0000 mon.smithi078 (mon.0) 3 : cluster [DBG] fsmap 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:44.204920+0000 mon.smithi078 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-21T03:38:44.591 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:44.205338+0000 mon.smithi078 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-12-21T03:38:44.614 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.612+0000 7f9c9083b700 1 -- 172.21.15.78:0/2102507395 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f9c8c101c50 con 0x7f9c8c072490 2023-12-21T03:38:44.618 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.617+0000 7f9c837fe700 1 -- 172.21.15.78:0/2102507395 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9c7c01bdd0 con 0x7f9c8c072490 2023-12-21T03:38:44.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.618+0000 7f9c9083b700 1 -- 172.21.15.78:0/2102507395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 msgr2=0x7f9c8c10cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:44.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.618+0000 7f9c9083b700 1 --2- 172.21.15.78:0/2102507395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c10cb00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f9c7c00b190 tx=0x7f9c7c006ee0).stop 2023-12-21T03:38:44.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.618+0000 7f9c9083b700 1 -- 172.21.15.78:0/2102507395 shutdown_connections 2023-12-21T03:38:44.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.618+0000 7f9c9083b700 1 --2- 172.21.15.78:0/2102507395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c8c072490 0x7f9c8c10cb00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:44.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.618+0000 7f9c9083b700 1 -- 172.21.15.78:0/2102507395 >> 172.21.15.78:0/2102507395 conn(0x7f9c8c0f7380 msgr2=0x7f9c8c0f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:44.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.618+0000 7f9c9083b700 1 -- 172.21.15.78:0/2102507395 shutdown_connections 2023-12-21T03:38:44.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:44.618+0000 7f9c9083b700 1 -- 172.21.15.78:0/2102507395 wait complete. 2023-12-21T03:38:44.996 INFO:teuthology.orchestra.run.smithi078.stderr:Wrote config to /etc/ceph/ceph.conf 2023-12-21T03:38:44.997 INFO:teuthology.orchestra.run.smithi078.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-21T03:38:44.997 INFO:teuthology.orchestra.run.smithi078.stderr:Creating mgr... 2023-12-21T03:38:44.997 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying port 9283 ... 2023-12-21T03:38:44.998 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl daemon-reload 2023-12-21T03:38:45.092 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl stop ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mgr.smithi078.zidbgq 2023-12-21T03:38:45.121 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl reset-failed ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mgr.smithi078.zidbgq 2023-12-21T03:38:45.127 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to reset failed state of unit ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mgr.smithi078.zidbgq.service: Unit ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mgr.smithi078.zidbgq.service not loaded. 2023-12-21T03:38:45.128 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl enable ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mgr.smithi078.zidbgq 2023-12-21T03:38:45.133 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Created symlink /etc/systemd/system/ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715.target.wants/ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mgr.smithi078.zidbgq.service → /etc/systemd/system/ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@.service. 2023-12-21T03:38:45.215 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mgr.smithi078.zidbgq 2023-12-21T03:38:45.567 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-21T03:38:45.576 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: disabled 2023-12-21T03:38:45.577 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active firewalld.service 2023-12-21T03:38:45.584 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-12-21T03:38:45.585 INFO:teuthology.orchestra.run.smithi078.stderr:firewalld.service is not enabled 2023-12-21T03:38:45.585 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to enable service . firewalld.service is not available 2023-12-21T03:38:45.585 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-21T03:38:45.585 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-21T03:38:45.592 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: disabled 2023-12-21T03:38:45.593 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active firewalld.service 2023-12-21T03:38:45.600 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-12-21T03:38:45.601 INFO:teuthology.orchestra.run.smithi078.stderr:firewalld.service is not enabled 2023-12-21T03:38:45.601 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-21T03:38:45.602 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mgr to start... 2023-12-21T03:38:45.602 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mgr... 2023-12-21T03:38:45.602 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-21T03:38:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:44.618284+0000 mon.smithi078 (mon.0) 6 : audit [INF] from='client.? 172.21.15.78:0/2102507395' entity='client.admin' 2023-12-21T03:38:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.961+0000 7f90f289d700 1 Processor -- start 2023-12-21T03:38:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.961+0000 7f90f289d700 1 -- start start 2023-12-21T03:38:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.962+0000 7f90f289d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.962+0000 7f90f289d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec0fedd0 con 0x7f90ec070430 2023-12-21T03:38:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.962+0000 7f90ebfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.962+0000 7f90ebfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50496/0 (socket says 172.21.15.78:50496) 2023-12-21T03:38:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.962+0000 7f90ebfff700 1 -- 172.21.15.78:0/2135631019 learned_addr learned my addr 172.21.15.78:0/2135631019 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.962+0000 7f90ebfff700 1 -- 172.21.15.78:0/2135631019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec0fef10 con 0x7f90ec070430 2023-12-21T03:38:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.963+0000 7f90ebfff700 1 --2- 172.21.15.78:0/2135631019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f90dc00b5f0 tx=0x7f90dc006ee0).ready entity=mon.0 client_cookie=83a7c35ee77a2767 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.963+0000 7f90eaffd700 1 -- 172.21.15.78:0/2135631019 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90dc00d600 con 0x7f90ec070430 2023-12-21T03:38:45.966 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.963+0000 7f90eaffd700 1 -- 172.21.15.78:0/2135631019 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f90dc00dbd0 con 0x7f90ec070430 2023-12-21T03:38:45.966 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.963+0000 7f90eaffd700 1 -- 172.21.15.78:0/2135631019 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90dc011c00 con 0x7f90ec070430 2023-12-21T03:38:45.967 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.964+0000 7f90f289d700 1 -- 172.21.15.78:0/2135631019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 msgr2=0x7f90ec0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:45.967 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.964+0000 7f90f289d700 1 --2- 172.21.15.78:0/2135631019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f90dc00b5f0 tx=0x7f90dc006ee0).stop 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.964+0000 7f90f289d700 1 -- 172.21.15.78:0/2135631019 shutdown_connections 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.964+0000 7f90f289d700 1 --2- 172.21.15.78:0/2135631019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fe8e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.964+0000 7f90f289d700 1 -- 172.21.15.78:0/2135631019 >> 172.21.15.78:0/2135631019 conn(0x7f90ec0f63c0 msgr2=0x7f90ec0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.964+0000 7f90f289d700 1 -- 172.21.15.78:0/2135631019 shutdown_connections 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.964+0000 7f90f289d700 1 -- 172.21.15.78:0/2135631019 wait complete. 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.964+0000 7f90f289d700 1 Processor -- start 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.965+0000 7f90f289d700 1 -- start start 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.965+0000 7f90f289d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.965+0000 7f90f289d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec0fa7e0 con 0x7f90ec070430 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.965+0000 7f90ebfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.965+0000 7f90ebfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50506/0 (socket says 172.21.15.78:50506) 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.965+0000 7f90ebfff700 1 -- 172.21.15.78:0/4009614599 learned_addr learned my addr 172.21.15.78:0/4009614599 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.965+0000 7f90ebfff700 1 -- 172.21.15.78:0/4009614599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc00b200 con 0x7f90ec070430 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.965+0000 7f90ebfff700 1 --2- 172.21.15.78:0/4009614599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f90dc00b190 tx=0x7f90dc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:45.968 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.966+0000 7f90e8ff9700 1 -- 172.21.15.78:0/4009614599 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90dc01b410 con 0x7f90ec070430 2023-12-21T03:38:45.969 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.966+0000 7f90e8ff9700 1 -- 172.21.15.78:0/4009614599 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f90dc01b9e0 con 0x7f90ec070430 2023-12-21T03:38:45.969 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.966+0000 7f90f289d700 1 -- 172.21.15.78:0/4009614599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ec0fa980 con 0x7f90ec070430 2023-12-21T03:38:45.970 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.966+0000 7f90e8ff9700 1 -- 172.21.15.78:0/4009614599 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90dc01a880 con 0x7f90ec070430 2023-12-21T03:38:45.970 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.966+0000 7f90f289d700 1 -- 172.21.15.78:0/4009614599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ec0fad90 con 0x7f90ec070430 2023-12-21T03:38:45.970 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.966+0000 7f90e8ff9700 1 -- 172.21.15.78:0/4009614599 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f90dc00f070 con 0x7f90ec070430 2023-12-21T03:38:45.970 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.966+0000 7f90e8ff9700 1 -- 172.21.15.78:0/4009614599 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f90dc025b70 con 0x7f90ec070430 2023-12-21T03:38:45.970 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.967+0000 7f90f289d700 1 -- 172.21.15.78:0/4009614599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ec045ad0 con 0x7f90ec070430 2023-12-21T03:38:45.970 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:45.969+0000 7f90e8ff9700 1 -- 172.21.15.78:0/4009614599 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f90dc016070 con 0x7f90ec070430 2023-12-21T03:38:46.044 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:46.036+0000 7f90f289d700 1 -- 172.21.15.78:0/4009614599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f90ec0008d0 con 0x7f90ec070430 2023-12-21T03:38:46.044 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:46.037+0000 7f90e8ff9700 1 -- 172.21.15.78:0/4009614599 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f90dc006a70 con 0x7f90ec070430 2023-12-21T03:38:46.044 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:46.044 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsid": "5ef0788c-9fb2-11ee-95a5-87774f69a715", 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "health": { 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "checks": {}, 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mutes": [] 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum": [ 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 0 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "smithi078" 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-12-21T03:38:46.045 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-12-21T03:38:46.046 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "monmap": { 2023-12-21T03:38:46.046 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:46.046 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-21T03:38:46.046 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-21T03:38:46.046 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:46.046 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osdmap": { 2023-12-21T03:38:46.046 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:46.046 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgmap": { 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-21T03:38:46.047 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsmap": { 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "available": false, 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modules": [ 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "iostat", 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "restful" 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-12-21T03:38:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "servicemap": { 2023-12-21T03:38:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modified": "2023-12-21T03:38:40.709470+0000", 2023-12-21T03:38:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-12-21T03:38:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-21T03:38:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-12-21T03:38:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:46.038+0000 7f90f289d700 1 -- 172.21.15.78:0/4009614599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 msgr2=0x7f90ec0fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:46.038+0000 7f90f289d700 1 --2- 172.21.15.78:0/4009614599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f90dc00b190 tx=0x7f90dc006ee0).stop 2023-12-21T03:38:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:46.038+0000 7f90f289d700 1 -- 172.21.15.78:0/4009614599 shutdown_connections 2023-12-21T03:38:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:46.038+0000 7f90f289d700 1 --2- 172.21.15.78:0/4009614599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90ec070430 0x7f90ec0fc360 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:46.038+0000 7f90f289d700 1 -- 172.21.15.78:0/4009614599 >> 172.21.15.78:0/4009614599 conn(0x7f90ec0f63c0 msgr2=0x7f90ec0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:46.038+0000 7f90f289d700 1 -- 172.21.15.78:0/4009614599 shutdown_connections 2023-12-21T03:38:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:46.038+0000 7f90f289d700 1 -- 172.21.15.78:0/4009614599 wait complete. 2023-12-21T03:38:46.412 INFO:teuthology.orchestra.run.smithi078.stderr:mgr not available, waiting (1/15)... 2023-12-21T03:38:47.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:46 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:46.038027+0000 mon.smithi078 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.78:0/4009614599' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-21T03:38:48.416 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-21T03:38:48.851 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.848+0000 7f5cd1bcb700 1 Processor -- start 2023-12-21T03:38:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.849+0000 7f5cd1bcb700 1 -- start start 2023-12-21T03:38:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.849+0000 7f5cd1bcb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.849+0000 7f5cd1bcb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0fedd0 con 0x7f5ccc070430 2023-12-21T03:38:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.849+0000 7f5ccb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.849+0000 7f5ccb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41794/0 (socket says 172.21.15.78:41794) 2023-12-21T03:38:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.849+0000 7f5ccb7fe700 1 -- 172.21.15.78:0/3996485411 learned_addr learned my addr 172.21.15.78:0/3996485411 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.849+0000 7f5ccb7fe700 1 -- 172.21.15.78:0/3996485411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc0fef10 con 0x7f5ccc070430 2023-12-21T03:38:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.849+0000 7f5ccb7fe700 1 --2- 172.21.15.78:0/3996485411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc0fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5cbc00b5f0 tx=0x7f5cbc006ee0).ready entity=mon.0 client_cookie=2f9f006f212fa13b server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.849+0000 7f5cca7fc700 1 -- 172.21.15.78:0/3996485411 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cbc00d600 con 0x7f5ccc070430 2023-12-21T03:38:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.850+0000 7f5cca7fc700 1 -- 172.21.15.78:0/3996485411 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5cbc00dbd0 con 0x7f5ccc070430 2023-12-21T03:38:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.850+0000 7f5cca7fc700 1 -- 172.21.15.78:0/3996485411 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cbc011c00 con 0x7f5ccc070430 2023-12-21T03:38:48.854 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.850+0000 7f5cd1bcb700 1 -- 172.21.15.78:0/3996485411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 msgr2=0x7f5ccc0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:48.854 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.850+0000 7f5cd1bcb700 1 --2- 172.21.15.78:0/3996485411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc0fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5cbc00b5f0 tx=0x7f5cbc006ee0).stop 2023-12-21T03:38:48.854 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.850+0000 7f5cd1bcb700 1 -- 172.21.15.78:0/3996485411 shutdown_connections 2023-12-21T03:38:48.854 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.850+0000 7f5cd1bcb700 1 --2- 172.21.15.78:0/3996485411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc0fe8e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:48.854 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.850+0000 7f5cd1bcb700 1 -- 172.21.15.78:0/3996485411 >> 172.21.15.78:0/3996485411 conn(0x7f5ccc0f63c0 msgr2=0x7f5ccc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:48.854 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.850+0000 7f5cd1bcb700 1 -- 172.21.15.78:0/3996485411 shutdown_connections 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.850+0000 7f5cd1bcb700 1 -- 172.21.15.78:0/3996485411 wait complete. 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.851+0000 7f5cd1bcb700 1 Processor -- start 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.851+0000 7f5cd1bcb700 1 -- start start 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.851+0000 7f5cd1bcb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.851+0000 7f5cd1bcb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc10aef0 con 0x7f5ccc070430 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.852+0000 7f5ccb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.852+0000 7f5ccb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41806/0 (socket says 172.21.15.78:41806) 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.852+0000 7f5ccb7fe700 1 -- 172.21.15.78:0/3348229639 learned_addr learned my addr 172.21.15.78:0/3348229639 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.852+0000 7f5ccb7fe700 1 -- 172.21.15.78:0/3348229639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc00b200 con 0x7f5ccc070430 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.852+0000 7f5ccb7fe700 1 --2- 172.21.15.78:0/3348229639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc10aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5cbc00b190 tx=0x7f5cbc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.852+0000 7f5cb3fff700 1 -- 172.21.15.78:0/3348229639 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cbc01b410 con 0x7f5ccc070430 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.852+0000 7f5cb3fff700 1 -- 172.21.15.78:0/3348229639 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5cbc01b9e0 con 0x7f5ccc070430 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.852+0000 7f5cd1bcb700 1 -- 172.21.15.78:0/3348229639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc10b0f0 con 0x7f5ccc070430 2023-12-21T03:38:48.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.852+0000 7f5cb3fff700 1 -- 172.21.15.78:0/3348229639 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cbc01a880 con 0x7f5ccc070430 2023-12-21T03:38:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.852+0000 7f5cd1bcb700 1 -- 172.21.15.78:0/3348229639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc10b500 con 0x7f5ccc070430 2023-12-21T03:38:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.853+0000 7f5cb1ffb700 1 -- 172.21.15.78:0/3348229639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca4004fa0 con 0x7f5ccc070430 2023-12-21T03:38:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.853+0000 7f5cb3fff700 1 -- 172.21.15.78:0/3348229639 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f5cbc00f070 con 0x7f5ccc070430 2023-12-21T03:38:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.853+0000 7f5cb3fff700 1 -- 172.21.15.78:0/3348229639 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5cbc01ae60 con 0x7f5ccc070430 2023-12-21T03:38:48.857 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.855+0000 7f5cb3fff700 1 -- 172.21.15.78:0/3348229639 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5cbc016070 con 0x7f5ccc070430 2023-12-21T03:38:48.933 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.925+0000 7f5cb1ffb700 1 -- 172.21.15.78:0/3348229639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5ca4005670 con 0x7f5ccc070430 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.926+0000 7f5cb3fff700 1 -- 172.21.15.78:0/3348229639 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f5cbc01b7d0 con 0x7f5ccc070430 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsid": "5ef0788c-9fb2-11ee-95a5-87774f69a715", 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "health": { 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "checks": {}, 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mutes": [] 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum": [ 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 0 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-12-21T03:38:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "smithi078" 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "monmap": { 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osdmap": { 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-21T03:38:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgmap": { 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsmap": { 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "available": false, 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modules": [ 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "iostat", 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "restful" 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "servicemap": { 2023-12-21T03:38:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modified": "2023-12-21T03:38:40.709470+0000", 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.926+0000 7f5cb1ffb700 1 -- 172.21.15.78:0/3348229639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 msgr2=0x7f5ccc10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.926+0000 7f5cb1ffb700 1 --2- 172.21.15.78:0/3348229639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc10aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5cbc00b190 tx=0x7f5cbc006ee0).stop 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.926+0000 7f5cb1ffb700 1 -- 172.21.15.78:0/3348229639 shutdown_connections 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.926+0000 7f5cb1ffb700 1 --2- 172.21.15.78:0/3348229639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc070430 0x7f5ccc10aa00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.926+0000 7f5cb1ffb700 1 -- 172.21.15.78:0/3348229639 >> 172.21.15.78:0/3348229639 conn(0x7f5ccc0f63c0 msgr2=0x7f5ccc0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.927+0000 7f5cb1ffb700 1 -- 172.21.15.78:0/3348229639 shutdown_connections 2023-12-21T03:38:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:48.927+0000 7f5cb1ffb700 1 -- 172.21.15.78:0/3348229639 wait complete. 2023-12-21T03:38:49.289 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:48.926903+0000 mon.smithi078 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.78:0/3348229639' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-21T03:38:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:mgr not available, waiting (2/15)... 2023-12-21T03:38:50.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:50.436954+0000 mon.smithi078 (mon.0) 9 : cluster [INF] Activating manager daemon smithi078.zidbgq 2023-12-21T03:38:50.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:50.439533+0000 mon.smithi078 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi078.zidbgq(active, starting, since 0.00269845s) 2023-12-21T03:38:50.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:50.440353+0000 mon.smithi078 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.78:0/1823556274' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-21T03:38:50.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:50.440538+0000 mon.smithi078 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.78:0/1823556274' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-21T03:38:50.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:50.440697+0000 mon.smithi078 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.78:0/1823556274' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-21T03:38:50.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:50.440861+0000 mon.smithi078 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.78:0/1823556274' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2023-12-21T03:38:50.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:50.441040+0000 mon.smithi078 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.78:0/1823556274' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mgr metadata", "who": "smithi078.zidbgq", "id": "smithi078.zidbgq"}]: dispatch 2023-12-21T03:38:50.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:50.454304+0000 mon.smithi078 (mon.0) 16 : cluster [INF] Manager daemon smithi078.zidbgq is now available 2023-12-21T03:38:50.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:50.461669+0000 mon.smithi078 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.78:0/1823556274' entity='mgr.smithi078.zidbgq' 2023-12-21T03:38:50.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:50.465861+0000 mon.smithi078 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.78:0/1823556274' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:38:50.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:50.466931+0000 mon.smithi078 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.78:0/1823556274' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:38:50.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:50.471312+0000 mon.smithi078 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.78:0/1823556274' entity='mgr.smithi078.zidbgq' 2023-12-21T03:38:50.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:50.473281+0000 mon.smithi078 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.78:0/1823556274' entity='mgr.smithi078.zidbgq' 2023-12-21T03:38:51.322 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-21T03:38:51.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.678+0000 7f0dd0dc7700 1 Processor -- start 2023-12-21T03:38:51.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.678+0000 7f0dd0dc7700 1 -- start start 2023-12-21T03:38:51.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.678+0000 7f0dd0dc7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:51.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.678+0000 7f0dd0dc7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc0fb490 con 0x7f0dcc0fab80 2023-12-21T03:38:51.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.679+0000 7f0dca59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:51.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.679+0000 7f0dca59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41866/0 (socket says 172.21.15.78:41866) 2023-12-21T03:38:51.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.679+0000 7f0dca59c700 1 -- 172.21.15.78:0/4255289113 learned_addr learned my addr 172.21.15.78:0/4255289113 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:51.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.679+0000 7f0dca59c700 1 -- 172.21.15.78:0/4255289113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc0fb5d0 con 0x7f0dcc0fab80 2023-12-21T03:38:51.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.679+0000 7f0dca59c700 1 --2- 172.21.15.78:0/4255289113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc0fafa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0db400b5f0 tx=0x7f0db4006ee0).ready entity=mon.0 client_cookie=7bb5623993b5a58 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:51.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.679+0000 7f0dc959a700 1 -- 172.21.15.78:0/4255289113 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0db400d600 con 0x7f0dcc0fab80 2023-12-21T03:38:51.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.679+0000 7f0dc959a700 1 -- 172.21.15.78:0/4255289113 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0db400dbd0 con 0x7f0dcc0fab80 2023-12-21T03:38:51.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.679+0000 7f0dc959a700 1 -- 172.21.15.78:0/4255289113 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0db4011c00 con 0x7f0dcc0fab80 2023-12-21T03:38:51.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.680+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/4255289113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 msgr2=0x7f0dcc0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:51.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.680+0000 7f0dd0dc7700 1 --2- 172.21.15.78:0/4255289113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc0fafa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0db400b5f0 tx=0x7f0db4006ee0).stop 2023-12-21T03:38:51.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.680+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/4255289113 shutdown_connections 2023-12-21T03:38:51.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.680+0000 7f0dd0dc7700 1 --2- 172.21.15.78:0/4255289113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc0fafa0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:51.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.680+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/4255289113 >> 172.21.15.78:0/4255289113 conn(0x7f0dcc0f63c0 msgr2=0x7f0dcc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:51.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.680+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/4255289113 shutdown_connections 2023-12-21T03:38:51.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.680+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/4255289113 wait complete. 2023-12-21T03:38:51.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dd0dc7700 1 Processor -- start 2023-12-21T03:38:51.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dd0dc7700 1 -- start start 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dd0dc7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dd0dc7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc0ff290 con 0x7f0dcc0fab80 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dca59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dca59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41880/0 (socket says 172.21.15.78:41880) 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dca59c700 1 -- 172.21.15.78:0/513164553 learned_addr learned my addr 172.21.15.78:0/513164553 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dca59c700 1 -- 172.21.15.78:0/513164553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db400b200 con 0x7f0dcc0fab80 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dca59c700 1 --2- 172.21.15.78:0/513164553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc100e10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f0db400e040 tx=0x7f0db4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dc2ffd700 1 -- 172.21.15.78:0/513164553 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0db401b410 con 0x7f0dcc0fab80 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/513164553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dcc0ff430 con 0x7f0dcc0fab80 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dc2ffd700 1 -- 172.21.15.78:0/513164553 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0db401b9e0 con 0x7f0dcc0fab80 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dc2ffd700 1 -- 172.21.15.78:0/513164553 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0db401a820 con 0x7f0dcc0fab80 2023-12-21T03:38:51.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.681+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/513164553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dcc0ff840 con 0x7f0dcc0fab80 2023-12-21T03:38:51.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.682+0000 7f0dc2ffd700 1 -- 172.21.15.78:0/513164553 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f0db400f070 con 0x7f0dcc0fab80 2023-12-21T03:38:51.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.682+0000 7f0dc2ffd700 1 --2- 172.21.15.78:0/513164553 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f0dac02fc50 0x7f0dac032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:51.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.682+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/513164553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dcc045ad0 con 0x7f0dcc0fab80 2023-12-21T03:38:51.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.682+0000 7f0dc2ffd700 1 -- 172.21.15.78:0/513164553 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0db40411e0 con 0x7f0dcc0fab80 2023-12-21T03:38:51.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.682+0000 7f0dc9d9b700 1 --2- 172.21.15.78:0/513164553 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f0dac02fc50 0x7f0dac032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:51.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.683+0000 7f0dc9d9b700 1 --2- 172.21.15.78:0/513164553 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f0dac02fc50 0x7f0dac032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0dbc0096a0 tx=0x7f0dbc006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:51.688 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.687+0000 7f0dc2ffd700 1 -- 172.21.15.78:0/513164553 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0db4016070 con 0x7f0dcc0fab80 2023-12-21T03:38:51.897 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.888+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/513164553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0dcc110a40 con 0x7f0dcc0fab80 2023-12-21T03:38:51.897 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.889+0000 7f0dc2ffd700 1 -- 172.21.15.78:0/513164553 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f0db4022b90 con 0x7f0dcc0fab80 2023-12-21T03:38:51.898 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.898 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-12-21T03:38:51.898 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsid": "5ef0788c-9fb2-11ee-95a5-87774f69a715", 2023-12-21T03:38:51.898 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "health": { 2023-12-21T03:38:51.898 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-21T03:38:51.898 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "checks": {}, 2023-12-21T03:38:51.898 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mutes": [] 2023-12-21T03:38:51.898 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum": [ 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 0 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "smithi078" 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "monmap": { 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-21T03:38:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osdmap": { 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgmap": { 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-21T03:38:51.900 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsmap": { 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:51.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "available": true, 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modules": [ 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "iostat", 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "restful" 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "servicemap": { 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modified": "2023-12-21T03:38:40.709470+0000", 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-12-21T03:38:51.902 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.890+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/513164553 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f0dac02fc50 msgr2=0x7f0dac032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.890+0000 7f0dd0dc7700 1 --2- 172.21.15.78:0/513164553 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f0dac02fc50 0x7f0dac032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0dbc0096a0 tx=0x7f0dbc006b40).stop 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.890+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/513164553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 msgr2=0x7f0dcc100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.890+0000 7f0dd0dc7700 1 --2- 172.21.15.78:0/513164553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc100e10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f0db400e040 tx=0x7f0db4006ee0).stop 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.890+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/513164553 shutdown_connections 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.890+0000 7f0dd0dc7700 1 --2- 172.21.15.78:0/513164553 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f0dac02fc50 0x7f0dac032110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.890+0000 7f0dd0dc7700 1 --2- 172.21.15.78:0/513164553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dcc0fab80 0x7f0dcc100e10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.890+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/513164553 >> 172.21.15.78:0/513164553 conn(0x7f0dcc0f63c0 msgr2=0x7f0dcc070030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.890+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/513164553 shutdown_connections 2023-12-21T03:38:51.903 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:51.890+0000 7f0dd0dc7700 1 -- 172.21.15.78:0/513164553 wait complete. 2023-12-21T03:38:52.261 INFO:teuthology.orchestra.run.smithi078.stderr:mgr is available 2023-12-21T03:38:52.261 INFO:teuthology.orchestra.run.smithi078.stderr:Enabling cephadm module... 2023-12-21T03:38:52.261 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-12-21T03:38:52.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:51.442412+0000 mon.smithi078 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi078.zidbgq(active, since 1.00558s) 2023-12-21T03:38:52.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:51.890112+0000 mon.smithi078 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.78:0/513164553' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-21T03:38:52.614 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.611+0000 7f4f0f589700 1 Processor -- start 2023-12-21T03:38:52.614 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.611+0000 7f4f0f589700 1 -- start start 2023-12-21T03:38:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.612+0000 7f4f0f589700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f08100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.612+0000 7f4f0f589700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080fc710 con 0x7f4f080703f0 2023-12-21T03:38:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.612+0000 7f4f0d325700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f08100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.612+0000 7f4f0d325700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f08100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41896/0 (socket says 172.21.15.78:41896) 2023-12-21T03:38:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.612+0000 7f4f0d325700 1 -- 172.21.15.78:0/3902667505 learned_addr learned my addr 172.21.15.78:0/3902667505 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.612+0000 7f4f0d325700 1 -- 172.21.15.78:0/3902667505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f08101210 con 0x7f4f080703f0 2023-12-21T03:38:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.612+0000 7f4f0d325700 1 --2- 172.21.15.78:0/3902667505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f08100d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f4ef800b5f0 tx=0x7f4ef8006ee0).ready entity=mon.0 client_cookie=70b3c71392d841fc server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.612+0000 7f4ef7fff700 1 -- 172.21.15.78:0/3902667505 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ef800d600 con 0x7f4f080703f0 2023-12-21T03:38:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.612+0000 7f4ef7fff700 1 -- 172.21.15.78:0/3902667505 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4ef800dbd0 con 0x7f4f080703f0 2023-12-21T03:38:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.612+0000 7f4ef7fff700 1 -- 172.21.15.78:0/3902667505 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ef8011c00 con 0x7f4f080703f0 2023-12-21T03:38:52.618 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.613+0000 7f4f0f589700 1 -- 172.21.15.78:0/3902667505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 msgr2=0x7f4f08100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:52.618 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.613+0000 7f4f0f589700 1 --2- 172.21.15.78:0/3902667505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f08100d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f4ef800b5f0 tx=0x7f4ef8006ee0).stop 2023-12-21T03:38:52.618 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.613+0000 7f4f0f589700 1 -- 172.21.15.78:0/3902667505 shutdown_connections 2023-12-21T03:38:52.618 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.613+0000 7f4f0f589700 1 --2- 172.21.15.78:0/3902667505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f08100d20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:52.618 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.613+0000 7f4f0f589700 1 -- 172.21.15.78:0/3902667505 >> 172.21.15.78:0/3902667505 conn(0x7f4f080f63c0 msgr2=0x7f4f080f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:52.618 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.613+0000 7f4f0f589700 1 -- 172.21.15.78:0/3902667505 shutdown_connections 2023-12-21T03:38:52.618 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.613+0000 7f4f0f589700 1 -- 172.21.15.78:0/3902667505 wait complete. 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.614+0000 7f4f0f589700 1 Processor -- start 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.614+0000 7f4f0f589700 1 -- start start 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.614+0000 7f4f0f589700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f0810cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.614+0000 7f4f0f589700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0810d0b0 con 0x7f4f080703f0 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.614+0000 7f4f0d325700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f0810cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.614+0000 7f4f0d325700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f0810cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41902/0 (socket says 172.21.15.78:41902) 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.614+0000 7f4f0d325700 1 -- 172.21.15.78:0/1683382926 learned_addr learned my addr 172.21.15.78:0/1683382926 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.614+0000 7f4f0d325700 1 -- 172.21.15.78:0/1683382926 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef800b200 con 0x7f4f080703f0 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.614+0000 7f4f0d325700 1 --2- 172.21.15.78:0/1683382926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f0810cbc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f4ef800b190 tx=0x7f4ef8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.614+0000 7f4ef5ffb700 1 -- 172.21.15.78:0/1683382926 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ef801b410 con 0x7f4f080703f0 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.615+0000 7f4ef5ffb700 1 -- 172.21.15.78:0/1683382926 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4ef801b9e0 con 0x7f4f080703f0 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.615+0000 7f4f0f589700 1 -- 172.21.15.78:0/1683382926 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0810d2b0 con 0x7f4f080703f0 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.615+0000 7f4ef5ffb700 1 -- 172.21.15.78:0/1683382926 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ef801a880 con 0x7f4f080703f0 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.615+0000 7f4f0f589700 1 -- 172.21.15.78:0/1683382926 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0810d6c0 con 0x7f4f080703f0 2023-12-21T03:38:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.616+0000 7f4ef5ffb700 1 -- 172.21.15.78:0/1683382926 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f4ef800f070 con 0x7f4f080703f0 2023-12-21T03:38:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.616+0000 7f4ef5ffb700 1 --2- 172.21.15.78:0/1683382926 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f4ef002fd50 0x7f4ef0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.616+0000 7f4ef5ffb700 1 -- 172.21.15.78:0/1683382926 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4ef8041090 con 0x7f4f080703f0 2023-12-21T03:38:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.616+0000 7f4f0f589700 1 -- 172.21.15.78:0/1683382926 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f08045ad0 con 0x7f4f080703f0 2023-12-21T03:38:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.616+0000 7f4f0cb24700 1 --2- 172.21.15.78:0/1683382926 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f4ef002fd50 0x7f4ef0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.616+0000 7f4f0cb24700 1 --2- 172.21.15.78:0/1683382926 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f4ef002fd50 0x7f4ef0032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4f040096a0 tx=0x7f4f04006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.620+0000 7f4ef5ffb700 1 -- 172.21.15.78:0/1683382926 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ef8016070 con 0x7f4f080703f0 2023-12-21T03:38:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:52.804+0000 7f4f0f589700 1 -- 172.21.15.78:0/1683382926 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f4f08106a20 con 0x7f4f080703f0 2023-12-21T03:38:53.450 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.448+0000 7f4ef5ffb700 1 -- 172.21.15.78:0/1683382926 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4ef801b570 con 0x7f4f080703f0 2023-12-21T03:38:53.450 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.449+0000 7f4ef5ffb700 1 -- 172.21.15.78:0/1683382926 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f4ef802e7a0 con 0x7f4f080703f0 2023-12-21T03:38:53.453 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.451+0000 7f4f0f589700 1 -- 172.21.15.78:0/1683382926 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f4ef002fd50 msgr2=0x7f4ef0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:53.453 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.451+0000 7f4f0f589700 1 --2- 172.21.15.78:0/1683382926 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f4ef002fd50 0x7f4ef0032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4f040096a0 tx=0x7f4f04006b40).stop 2023-12-21T03:38:53.453 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.451+0000 7f4f0f589700 1 -- 172.21.15.78:0/1683382926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 msgr2=0x7f4f0810cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:53.453 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.451+0000 7f4f0f589700 1 --2- 172.21.15.78:0/1683382926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f0810cbc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f4ef800b190 tx=0x7f4ef8006ee0).stop 2023-12-21T03:38:53.454 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.451+0000 7f4f0f589700 1 -- 172.21.15.78:0/1683382926 shutdown_connections 2023-12-21T03:38:53.454 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.451+0000 7f4f0f589700 1 --2- 172.21.15.78:0/1683382926 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f4ef002fd50 0x7f4ef0032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:53.454 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.451+0000 7f4f0f589700 1 --2- 172.21.15.78:0/1683382926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080703f0 0x7f4f0810cbc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:53.454 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.451+0000 7f4f0f589700 1 -- 172.21.15.78:0/1683382926 >> 172.21.15.78:0/1683382926 conn(0x7f4f080f63c0 msgr2=0x7f4f080f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:53.454 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.451+0000 7f4f0f589700 1 -- 172.21.15.78:0/1683382926 shutdown_connections 2023-12-21T03:38:53.454 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:53.451+0000 7f4f0f589700 1 -- 172.21.15.78:0/1683382926 wait complete. 2023-12-21T03:38:53.755 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:52.446396+0000 mon.smithi078 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi078.zidbgq(active, since 2s) 2023-12-21T03:38:53.755 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:52.806071+0000 mon.smithi078 (mon.0) 25 : audit [INF] from='client.? 172.21.15.78:0/1683382926' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-21T03:38:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-21T03:38:54.213 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.208+0000 7f87c7265700 1 Processor -- start 2023-12-21T03:38:54.213 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.208+0000 7f87c7265700 1 -- start start 2023-12-21T03:38:54.213 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.208+0000 7f87c7265700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:54.213 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.208+0000 7f87c7265700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00fee10 con 0x7f87c00703f0 2023-12-21T03:38:54.213 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.209+0000 7f87c5001700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:54.213 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.209+0000 7f87c5001700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41932/0 (socket says 172.21.15.78:41932) 2023-12-21T03:38:54.213 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.209+0000 7f87c5001700 1 -- 172.21.15.78:0/1529523395 learned_addr learned my addr 172.21.15.78:0/1529523395 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:54.213 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.209+0000 7f87c5001700 1 -- 172.21.15.78:0/1529523395 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c00fef50 con 0x7f87c00703f0 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.209+0000 7f87c5001700 1 --2- 172.21.15.78:0/1529523395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fe920 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f87bc00b5f0 tx=0x7f87bc006ee0).ready entity=mon.0 client_cookie=a992a4408e733746 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.209+0000 7f87b7fff700 1 -- 172.21.15.78:0/1529523395 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87bc00d600 con 0x7f87c00703f0 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.209+0000 7f87b7fff700 1 -- 172.21.15.78:0/1529523395 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f87bc00dbd0 con 0x7f87c00703f0 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.209+0000 7f87b7fff700 1 -- 172.21.15.78:0/1529523395 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87bc011c00 con 0x7f87c00703f0 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.210+0000 7f87c7265700 1 -- 172.21.15.78:0/1529523395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 msgr2=0x7f87c00fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.210+0000 7f87c7265700 1 --2- 172.21.15.78:0/1529523395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fe920 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f87bc00b5f0 tx=0x7f87bc006ee0).stop 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.210+0000 7f87c7265700 1 -- 172.21.15.78:0/1529523395 shutdown_connections 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.210+0000 7f87c7265700 1 --2- 172.21.15.78:0/1529523395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fe920 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.210+0000 7f87c7265700 1 -- 172.21.15.78:0/1529523395 >> 172.21.15.78:0/1529523395 conn(0x7f87c00f63c0 msgr2=0x7f87c00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.210+0000 7f87c7265700 1 -- 172.21.15.78:0/1529523395 shutdown_connections 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.210+0000 7f87c7265700 1 -- 172.21.15.78:0/1529523395 wait complete. 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.211+0000 7f87c7265700 1 Processor -- start 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.211+0000 7f87c7265700 1 -- start start 2023-12-21T03:38:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.211+0000 7f87c7265700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.211+0000 7f87c7265700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00fa820 con 0x7f87c00703f0 2023-12-21T03:38:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.211+0000 7f87c5001700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.211+0000 7f87c5001700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41946/0 (socket says 172.21.15.78:41946) 2023-12-21T03:38:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.211+0000 7f87c5001700 1 -- 172.21.15.78:0/799202210 learned_addr learned my addr 172.21.15.78:0/799202210 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.211+0000 7f87c5001700 1 -- 172.21.15.78:0/799202210 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc00b200 con 0x7f87c00703f0 2023-12-21T03:38:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.211+0000 7f87c5001700 1 --2- 172.21.15.78:0/799202210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fc3a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f87bc00b190 tx=0x7f87bc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.212+0000 7f87b5ffb700 1 -- 172.21.15.78:0/799202210 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87bc01b410 con 0x7f87c00703f0 2023-12-21T03:38:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.212+0000 7f87b5ffb700 1 -- 172.21.15.78:0/799202210 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f87bc01b9e0 con 0x7f87c00703f0 2023-12-21T03:38:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.212+0000 7f87c7265700 1 -- 172.21.15.78:0/799202210 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c00fa9c0 con 0x7f87c00703f0 2023-12-21T03:38:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.212+0000 7f87b5ffb700 1 -- 172.21.15.78:0/799202210 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87bc01a880 con 0x7f87c00703f0 2023-12-21T03:38:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.212+0000 7f87c7265700 1 -- 172.21.15.78:0/799202210 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c00fadd0 con 0x7f87c00703f0 2023-12-21T03:38:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.213+0000 7f87b5ffb700 1 -- 172.21.15.78:0/799202210 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f87bc00f070 con 0x7f87c00703f0 2023-12-21T03:38:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.213+0000 7f87b5ffb700 1 --2- 172.21.15.78:0/799202210 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f87a402fdb0 0x7f87a4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.213+0000 7f87b5ffb700 1 -- 172.21.15.78:0/799202210 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f87bc041fc0 con 0x7f87c00703f0 2023-12-21T03:38:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.213+0000 7f87c4800700 1 -- 172.21.15.78:0/799202210 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f87a402fdb0 msgr2=0x7f87a4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3728780489 2023-12-21T03:38:54.217 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.213+0000 7f87c4800700 1 --2- 172.21.15.78:0/799202210 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f87a402fdb0 0x7f87a4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-21T03:38:54.217 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.213+0000 7f87c7265700 1 -- 172.21.15.78:0/799202210 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c0059100 con 0x7f87c00703f0 2023-12-21T03:38:54.219 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.217+0000 7f87b5ffb700 1 -- 172.21.15.78:0/799202210 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87bc025460 con 0x7f87c00703f0 2023-12-21T03:38:54.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.410+0000 7f87c7265700 1 -- 172.21.15.78:0/799202210 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f87c00ffbd0 con 0x7f87c00703f0 2023-12-21T03:38:54.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.410+0000 7f87b5ffb700 1 -- 172.21.15.78:0/799202210 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f87bc006a70 con 0x7f87c00703f0 2023-12-21T03:38:54.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-12-21T03:38:54.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 5, 2023-12-21T03:38:54.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "available": true, 2023-12-21T03:38:54.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "active_name": "smithi078.zidbgq", 2023-12-21T03:38:54.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-21T03:38:54.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-12-21T03:38:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.412+0000 7f87c7265700 1 -- 172.21.15.78:0/799202210 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f87a402fdb0 msgr2=0x7f87a4032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-21T03:38:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.412+0000 7f87c7265700 1 --2- 172.21.15.78:0/799202210 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f87a402fdb0 0x7f87a4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:38:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.412+0000 7f87c7265700 1 -- 172.21.15.78:0/799202210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 msgr2=0x7f87c00fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.412+0000 7f87c7265700 1 --2- 172.21.15.78:0/799202210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fc3a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f87bc00b190 tx=0x7f87bc006ee0).stop 2023-12-21T03:38:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.412+0000 7f87c7265700 1 -- 172.21.15.78:0/799202210 shutdown_connections 2023-12-21T03:38:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.412+0000 7f87c7265700 1 --2- 172.21.15.78:0/799202210 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f87a402fdb0 0x7f87a4032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:38:54.416 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.412+0000 7f87c7265700 1 --2- 172.21.15.78:0/799202210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c00703f0 0x7f87c00fc3a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:54.416 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.412+0000 7f87c7265700 1 -- 172.21.15.78:0/799202210 >> 172.21.15.78:0/799202210 conn(0x7f87c00f63c0 msgr2=0x7f87c00f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:54.416 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.412+0000 7f87c7265700 1 -- 172.21.15.78:0/799202210 shutdown_connections 2023-12-21T03:38:54.416 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:54.412+0000 7f87c7265700 1 -- 172.21.15.78:0/799202210 wait complete. 2023-12-21T03:38:54.698 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:54 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:53.450090+0000 mon.smithi078 (mon.0) 26 : audit [INF] from='client.? 172.21.15.78:0/1683382926' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-21T03:38:54.698 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:54 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:53.450158+0000 mon.smithi078 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi078.zidbgq(active, since 3s) 2023-12-21T03:38:54.698 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:54 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:54.411797+0000 mon.smithi078 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.78:0/799202210' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-21T03:38:54.754 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for the mgr to restart... 2023-12-21T03:38:54.754 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mgr epoch 5... 2023-12-21T03:38:54.754 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-21T03:38:55.119 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.116+0000 7f985ce61700 1 Processor -- start 2023-12-21T03:38:55.119 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.116+0000 7f985ce61700 1 -- start start 2023-12-21T03:38:55.119 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.116+0000 7f985ce61700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:55.119 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.116+0000 7f985ce61700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98580fedd0 con 0x7f9858070430 2023-12-21T03:38:55.119 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.116+0000 7f985659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:55.119 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.116+0000 7f985659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41952/0 (socket says 172.21.15.78:41952) 2023-12-21T03:38:55.119 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.116+0000 7f985659c700 1 -- 172.21.15.78:0/860967227 learned_addr learned my addr 172.21.15.78:0/860967227 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:55.119 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.117+0000 7f985659c700 1 -- 172.21.15.78:0/860967227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98580fef10 con 0x7f9858070430 2023-12-21T03:38:55.119 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.117+0000 7f985659c700 1 --2- 172.21.15.78:0/860967227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f984800b5f0 tx=0x7f9848006ee0).ready entity=mon.0 client_cookie=fd77fbddaa953e4e server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:55.119 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.117+0000 7f985559a700 1 -- 172.21.15.78:0/860967227 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f984800d600 con 0x7f9858070430 2023-12-21T03:38:55.120 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.117+0000 7f985559a700 1 -- 172.21.15.78:0/860967227 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f984800dbd0 con 0x7f9858070430 2023-12-21T03:38:55.120 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.117+0000 7f985559a700 1 -- 172.21.15.78:0/860967227 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9848011c00 con 0x7f9858070430 2023-12-21T03:38:55.122 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.118+0000 7f985ce61700 1 -- 172.21.15.78:0/860967227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 msgr2=0x7f98580fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:55.122 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.118+0000 7f985ce61700 1 --2- 172.21.15.78:0/860967227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f984800b5f0 tx=0x7f9848006ee0).stop 2023-12-21T03:38:55.122 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.118+0000 7f985ce61700 1 -- 172.21.15.78:0/860967227 shutdown_connections 2023-12-21T03:38:55.122 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.118+0000 7f985ce61700 1 --2- 172.21.15.78:0/860967227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fe8e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:55.122 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.118+0000 7f985ce61700 1 -- 172.21.15.78:0/860967227 >> 172.21.15.78:0/860967227 conn(0x7f98580f63c0 msgr2=0x7f98580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:55.122 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.118+0000 7f985ce61700 1 -- 172.21.15.78:0/860967227 shutdown_connections 2023-12-21T03:38:55.122 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.118+0000 7f985ce61700 1 -- 172.21.15.78:0/860967227 wait complete. 2023-12-21T03:38:55.122 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.118+0000 7f985ce61700 1 Processor -- start 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.118+0000 7f985ce61700 1 -- start start 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.118+0000 7f985ce61700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.118+0000 7f985ce61700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98580fa7e0 con 0x7f9858070430 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.119+0000 7f985659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.119+0000 7f985659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41960/0 (socket says 172.21.15.78:41960) 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.119+0000 7f985659c700 1 -- 172.21.15.78:0/2373467454 learned_addr learned my addr 172.21.15.78:0/2373467454 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.119+0000 7f985659c700 1 -- 172.21.15.78:0/2373467454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984800b200 con 0x7f9858070430 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.119+0000 7f985659c700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fc360 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f984800e040 tx=0x7f9848006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.119+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f984800d9f0 con 0x7f9858070430 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.119+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f984801a9f0 con 0x7f9858070430 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.119+0000 7f985ce61700 1 -- 172.21.15.78:0/2373467454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98580fa980 con 0x7f9858070430 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.119+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f984801b8c0 con 0x7f9858070430 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.119+0000 7f985ce61700 1 -- 172.21.15.78:0/2373467454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98580fad90 con 0x7f9858070430 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.120+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f984800f070 con 0x7f9858070430 2023-12-21T03:38:55.123 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.120+0000 7f9846ffd700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 0x7f983c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:55.124 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.120+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 --> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f983c0328d0 con 0x7f983c02fd50 2023-12-21T03:38:55.124 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.120+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9848041260 con 0x7f9858070430 2023-12-21T03:38:55.124 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.120+0000 7f9855d9b700 1 -- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 msgr2=0x7f983c032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3728780489 2023-12-21T03:38:55.124 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.120+0000 7f9855d9b700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 0x7f983c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-21T03:38:55.323 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.321+0000 7f9855d9b700 1 -- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 msgr2=0x7f983c032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3728780489 2023-12-21T03:38:55.323 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.321+0000 7f9855d9b700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 0x7f983c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-21T03:38:55.724 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.722+0000 7f9855d9b700 1 -- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 msgr2=0x7f983c032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3728780489 2023-12-21T03:38:55.724 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:55.722+0000 7f9855d9b700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 0x7f983c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-21T03:38:56.525 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:56.523+0000 7f9855d9b700 1 -- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 msgr2=0x7f983c032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3728780489 2023-12-21T03:38:56.525 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:56.523+0000 7f9855d9b700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 0x7f983c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-21T03:38:58.127 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:58.124+0000 7f9855d9b700 1 -- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 msgr2=0x7f983c032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3728780489 2023-12-21T03:38:58.127 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:58.124+0000 7f9855d9b700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 0x7f983c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-21T03:38:58.429 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:58.427+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f984801ba20 con 0x7f9858070430 2023-12-21T03:38:58.429 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:58.427+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 msgr2=0x7f983c032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-21T03:38:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:58.427+0000 7f9846ffd700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 0x7f983c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:38:58.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:58 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:58.374034+0000 mon.smithi078 (mon.0) 29 : cluster [INF] Active manager daemon smithi078.zidbgq restarted 2023-12-21T03:38:58.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:58 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:58.374421+0000 mon.smithi078 (mon.0) 30 : cluster [INF] Activating manager daemon smithi078.zidbgq 2023-12-21T03:38:58.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:58 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:58.378667+0000 mon.smithi078 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-12-21T03:38:59.432 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.430+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f9848042bd0 con 0x7f9858070430 2023-12-21T03:38:59.432 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.430+0000 7f9846ffd700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f983c033870 0x7f983c035c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:38:59.432 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.430+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 --> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f983c0328d0 con 0x7f983c033870 2023-12-21T03:38:59.432 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.430+0000 7f9855d9b700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f983c033870 0x7f983c035c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:38:59.432 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.431+0000 7f9855d9b700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f983c033870 0x7f983c035c60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f984c006fd0 tx=0x7f984c006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:38:59.433 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.432+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 <== mgr.14114 v2:172.21.15.78:6800/1548863549 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f983c0328d0 con 0x7f983c033870 2023-12-21T03:38:59.443 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f985ce61700 1 -- 172.21.15.78:0/2373467454 --> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9858059100 con 0x7f983c033870 2023-12-21T03:38:59.443 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f9846ffd700 1 -- 172.21.15.78:0/2373467454 <== mgr.14114 v2:172.21.15.78:6800/1548863549 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9858059100 con 0x7f983c033870 2023-12-21T03:38:59.443 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-12-21T03:38:59.443 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-12-21T03:38:59.443 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "initialized": true 2023-12-21T03:38:59.443 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-12-21T03:38:59.443 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f985ce61700 1 -- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f983c033870 msgr2=0x7f983c035c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:59.443 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f985ce61700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f983c033870 0x7f983c035c60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f984c006fd0 tx=0x7f984c006cf0).stop 2023-12-21T03:38:59.443 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f985ce61700 1 -- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 msgr2=0x7f98580fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:38:59.443 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f985ce61700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fc360 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f984800e040 tx=0x7f9848006ee0).stop 2023-12-21T03:38:59.444 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f985ce61700 1 -- 172.21.15.78:0/2373467454 shutdown_connections 2023-12-21T03:38:59.444 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f985ce61700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9858070430 0x7f98580fc360 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:59.444 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f985ce61700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f983c033870 0x7f983c035c60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:38:59.444 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f985ce61700 1 --2- 172.21.15.78:0/2373467454 >> [v2:172.21.15.78:6800/3728780489,v1:172.21.15.78:6801/3728780489] conn(0x7f983c02fd50 0x7f983c032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:38:59.444 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f985ce61700 1 -- 172.21.15.78:0/2373467454 >> 172.21.15.78:0/2373467454 conn(0x7f98580f63c0 msgr2=0x7f98580f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:38:59.444 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.440+0000 7f985ce61700 1 -- 172.21.15.78:0/2373467454 shutdown_connections 2023-12-21T03:38:59.444 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:38:59.441+0000 7f985ce61700 1 -- 172.21.15.78:0/2373467454 wait complete. 2023-12-21T03:38:59.796 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:58.429865+0000 mon.smithi078 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi078.zidbgq(active, starting, since 0.0555403s) 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.431257+0000 mon.smithi078 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.431838+0000 mon.smithi078 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mgr metadata", "who": "smithi078.zidbgq", "id": "smithi078.zidbgq"}]: dispatch 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.432497+0000 mon.smithi078 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.432634+0000 mon.smithi078 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.432769+0000 mon.smithi078 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:58.434552+0000 mon.smithi078 (mon.0) 38 : cluster [INF] Manager daemon smithi078.zidbgq is now available 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.447491+0000 mon.smithi078 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.450217+0000 mon.smithi078 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.452628+0000 mon.smithi078 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.540254+0000 mon.smithi078 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.541026+0000 mon.smithi078 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:38:59.797 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.542603+0000 mon.smithi078 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:38:59.798 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.544105+0000 mon.smithi078 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:38:59.798 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.554109+0000 mon.smithi078 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:38:59.798 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:58.555239+0000 mon.smithi078 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:38:59.798 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:38:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:38:59.432552+0000 mon.smithi078 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi078.zidbgq(active, since 1.05822s) 2023-12-21T03:38:59.853 INFO:teuthology.orchestra.run.smithi078.stderr:mgr epoch 5 is available 2023-12-21T03:38:59.854 INFO:teuthology.orchestra.run.smithi078.stderr:Setting orchestrator backend to cephadm... 2023-12-21T03:38:59.854 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-12-21T03:39:00.224 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.223+0000 7f89fad9d700 1 Processor -- start 2023-12-21T03:39:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.223+0000 7f89fad9d700 1 -- start start 2023-12-21T03:39:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.223+0000 7f89fad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.224+0000 7f89fad9d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f40fb490 con 0x7f89f40fab80 2023-12-21T03:39:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.224+0000 7f89f8b39700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.224+0000 7f89f8b39700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39198/0 (socket says 172.21.15.78:39198) 2023-12-21T03:39:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.224+0000 7f89f8b39700 1 -- 172.21.15.78:0/3735896584 learned_addr learned my addr 172.21.15.78:0/3735896584 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.224+0000 7f89f8b39700 1 -- 172.21.15.78:0/3735896584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f40fb5d0 con 0x7f89f40fab80 2023-12-21T03:39:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.224+0000 7f89f8b39700 1 --2- 172.21.15.78:0/3735896584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f40fafa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f89e800b5f0 tx=0x7f89e8006ee0).ready entity=mon.0 client_cookie=84cc58f7749ab1ce server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.224+0000 7f89f37fe700 1 -- 172.21.15.78:0/3735896584 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89e800d600 con 0x7f89f40fab80 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.224+0000 7f89f37fe700 1 -- 172.21.15.78:0/3735896584 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f89e800dbd0 con 0x7f89f40fab80 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.224+0000 7f89f37fe700 1 -- 172.21.15.78:0/3735896584 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89e8011c00 con 0x7f89f40fab80 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.225+0000 7f89fad9d700 1 -- 172.21.15.78:0/3735896584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 msgr2=0x7f89f40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.225+0000 7f89fad9d700 1 --2- 172.21.15.78:0/3735896584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f40fafa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f89e800b5f0 tx=0x7f89e8006ee0).stop 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.225+0000 7f89fad9d700 1 -- 172.21.15.78:0/3735896584 shutdown_connections 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.225+0000 7f89fad9d700 1 --2- 172.21.15.78:0/3735896584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f40fafa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.225+0000 7f89fad9d700 1 -- 172.21.15.78:0/3735896584 >> 172.21.15.78:0/3735896584 conn(0x7f89f40f63c0 msgr2=0x7f89f40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.225+0000 7f89fad9d700 1 -- 172.21.15.78:0/3735896584 shutdown_connections 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.225+0000 7f89fad9d700 1 -- 172.21.15.78:0/3735896584 wait complete. 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.226+0000 7f89fad9d700 1 Processor -- start 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.226+0000 7f89fad9d700 1 -- start start 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.226+0000 7f89fad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f410fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.226+0000 7f89fad9d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f410b4f0 con 0x7f89f40fab80 2023-12-21T03:39:00.229 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.226+0000 7f89f8b39700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f410fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:00.230 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.226+0000 7f89f8b39700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f410fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39200/0 (socket says 172.21.15.78:39200) 2023-12-21T03:39:00.230 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.226+0000 7f89f8b39700 1 -- 172.21.15.78:0/3687096879 learned_addr learned my addr 172.21.15.78:0/3687096879 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:00.230 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.226+0000 7f89f8b39700 1 -- 172.21.15.78:0/3687096879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e800b200 con 0x7f89f40fab80 2023-12-21T03:39:00.230 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.226+0000 7f89f8b39700 1 --2- 172.21.15.78:0/3687096879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f410fb00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f89e800b1d0 tx=0x7f89e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:00.230 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.227+0000 7f89f17fa700 1 -- 172.21.15.78:0/3687096879 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89e801b410 con 0x7f89f40fab80 2023-12-21T03:39:00.230 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.227+0000 7f89fad9d700 1 -- 172.21.15.78:0/3687096879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f4070450 con 0x7f89f40fab80 2023-12-21T03:39:00.230 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.227+0000 7f89f17fa700 1 -- 172.21.15.78:0/3687096879 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f89e801b9e0 con 0x7f89f40fab80 2023-12-21T03:39:00.230 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.227+0000 7f89f17fa700 1 -- 172.21.15.78:0/3687096879 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89e801a840 con 0x7f89f40fab80 2023-12-21T03:39:00.230 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.227+0000 7f89fad9d700 1 -- 172.21.15.78:0/3687096879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f410fff0 con 0x7f89f40fab80 2023-12-21T03:39:00.231 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.227+0000 7f89f17fa700 1 -- 172.21.15.78:0/3687096879 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f89e800f070 con 0x7f89f40fab80 2023-12-21T03:39:00.231 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.227+0000 7f89f17fa700 1 --2- 172.21.15.78:0/3687096879 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f89d802fca0 0x7f89d8032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:00.231 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.228+0000 7f89f17fa700 1 -- 172.21.15.78:0/3687096879 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f89e80412b0 con 0x7f89f40fab80 2023-12-21T03:39:00.231 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.228+0000 7f89f3fff700 1 --2- 172.21.15.78:0/3687096879 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f89d802fca0 0x7f89d8032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:00.231 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.228+0000 7f89f3fff700 1 --2- 172.21.15.78:0/3687096879 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f89d802fca0 0x7f89d8032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f89e00096a0 tx=0x7f89e0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:00.231 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.228+0000 7f89fad9d700 1 -- 172.21.15.78:0/3687096879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f4045ad0 con 0x7f89f40fab80 2023-12-21T03:39:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.232+0000 7f89f17fa700 1 -- 172.21.15.78:0/3687096879 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89e8016030 con 0x7f89f40fab80 2023-12-21T03:39:00.415 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.414+0000 7f89fad9d700 1 -- 172.21.15.78:0/3687096879 --> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f89f40fc9a0 con 0x7f89d802fca0 2023-12-21T03:39:00.420 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.419+0000 7f89f17fa700 1 -- 172.21.15.78:0/3687096879 <== mgr.14114 v2:172.21.15.78:6800/1548863549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f89f40fc9a0 con 0x7f89d802fca0 2023-12-21T03:39:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.421+0000 7f89fad9d700 1 -- 172.21.15.78:0/3687096879 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f89d802fca0 msgr2=0x7f89d8032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.421+0000 7f89fad9d700 1 --2- 172.21.15.78:0/3687096879 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f89d802fca0 0x7f89d8032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f89e00096a0 tx=0x7f89e0006b40).stop 2023-12-21T03:39:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.421+0000 7f89fad9d700 1 -- 172.21.15.78:0/3687096879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 msgr2=0x7f89f410fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.421+0000 7f89fad9d700 1 --2- 172.21.15.78:0/3687096879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f410fb00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f89e800b1d0 tx=0x7f89e8006ee0).stop 2023-12-21T03:39:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.422+0000 7f89fad9d700 1 -- 172.21.15.78:0/3687096879 shutdown_connections 2023-12-21T03:39:00.424 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.422+0000 7f89fad9d700 1 --2- 172.21.15.78:0/3687096879 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f89d802fca0 0x7f89d8032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:00.424 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.422+0000 7f89fad9d700 1 --2- 172.21.15.78:0/3687096879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f40fab80 0x7f89f410fb00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:00.424 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.422+0000 7f89fad9d700 1 -- 172.21.15.78:0/3687096879 >> 172.21.15.78:0/3687096879 conn(0x7f89f40f63c0 msgr2=0x7f89f40f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:00.424 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.422+0000 7f89fad9d700 1 -- 172.21.15.78:0/3687096879 shutdown_connections 2023-12-21T03:39:00.424 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:00.422+0000 7f89fad9d700 1 -- 172.21.15.78:0/3687096879 wait complete. 2023-12-21T03:39:00.704 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:59.431743+0000 mgr.smithi078.zidbgq (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-21T03:39:00.704 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:38:59.440352+0000 mgr.smithi078.zidbgq (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-21T03:39:00.704 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:00.419464+0000 mon.smithi078 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:00.704 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:00.422825+0000 mon.smithi078 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:39:00.769 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-12-21T03:39:01.133 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.127+0000 7ff9824d8700 1 Processor -- start 2023-12-21T03:39:01.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.127+0000 7ff9824d8700 1 -- start start 2023-12-21T03:39:01.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.127+0000 7ff9824d8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:01.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.127+0000 7ff9824d8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c0fc5c0 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.127+0000 7ff97bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:01.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.128+0000 7ff97bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39204/0 (socket says 172.21.15.78:39204) 2023-12-21T03:39:01.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.128+0000 7ff97bfff700 1 -- 172.21.15.78:0/961177553 learned_addr learned my addr 172.21.15.78:0/961177553 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:01.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.128+0000 7ff97bfff700 1 -- 172.21.15.78:0/961177553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c0fc700 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.128+0000 7ff97bfff700 1 --2- 172.21.15.78:0/961177553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0fc0d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ff96c00b5f0 tx=0x7ff96c006ee0).ready entity=mon.0 client_cookie=1d62ae6c7dc850a8 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:01.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.128+0000 7ff97affd700 1 -- 172.21.15.78:0/961177553 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff96c00d600 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.128+0000 7ff97affd700 1 -- 172.21.15.78:0/961177553 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff96c00dbd0 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.128+0000 7ff97affd700 1 -- 172.21.15.78:0/961177553 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff96c011c00 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.129+0000 7ff9824d8700 1 -- 172.21.15.78:0/961177553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 msgr2=0x7ff97c0fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:01.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.129+0000 7ff9824d8700 1 --2- 172.21.15.78:0/961177553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0fc0d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ff96c00b5f0 tx=0x7ff96c006ee0).stop 2023-12-21T03:39:01.136 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.129+0000 7ff9824d8700 1 -- 172.21.15.78:0/961177553 shutdown_connections 2023-12-21T03:39:01.136 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.129+0000 7ff9824d8700 1 --2- 172.21.15.78:0/961177553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0fc0d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:01.136 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.129+0000 7ff9824d8700 1 -- 172.21.15.78:0/961177553 >> 172.21.15.78:0/961177553 conn(0x7ff97c0f74b0 msgr2=0x7ff97c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:01.136 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.129+0000 7ff9824d8700 1 -- 172.21.15.78:0/961177553 shutdown_connections 2023-12-21T03:39:01.136 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.129+0000 7ff9824d8700 1 -- 172.21.15.78:0/961177553 wait complete. 2023-12-21T03:39:01.136 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.129+0000 7ff9824d8700 1 Processor -- start 2023-12-21T03:39:01.136 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.129+0000 7ff9824d8700 1 -- start start 2023-12-21T03:39:01.136 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff9824d8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:01.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff9824d8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c070a60 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff97bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:01.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff97bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39212/0 (socket says 172.21.15.78:39212) 2023-12-21T03:39:01.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff97bfff700 1 -- 172.21.15.78:0/188048054 learned_addr learned my addr 172.21.15.78:0/188048054 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:01.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff97bfff700 1 -- 172.21.15.78:0/188048054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96c00b200 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff97bfff700 1 --2- 172.21.15.78:0/188048054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0725e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff96c00b1d0 tx=0x7ff96c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:01.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff978ff9700 1 -- 172.21.15.78:0/188048054 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff96c01b410 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff978ff9700 1 -- 172.21.15.78:0/188048054 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff96c01b9e0 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff9824d8700 1 -- 172.21.15.78:0/188048054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97c070c00 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff978ff9700 1 -- 172.21.15.78:0/188048054 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff96c01a840 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.130+0000 7ff9824d8700 1 -- 172.21.15.78:0/188048054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97c071010 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.131+0000 7ff978ff9700 1 -- 172.21.15.78:0/188048054 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff96c00f070 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.131+0000 7ff978ff9700 1 --2- 172.21.15.78:0/188048054 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7ff95c02fdb0 0x7ff95c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:01.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.131+0000 7ff978ff9700 1 -- 172.21.15.78:0/188048054 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff96c0411d0 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.131+0000 7ff97b7fe700 1 --2- 172.21.15.78:0/188048054 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7ff95c02fdb0 0x7ff95c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:01.139 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.132+0000 7ff9824d8700 1 -- 172.21.15.78:0/188048054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff97c059100 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.139 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.132+0000 7ff97b7fe700 1 --2- 172.21.15.78:0/188048054 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7ff95c02fdb0 0x7ff95c032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff9640096a0 tx=0x7ff964006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:01.140 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.136+0000 7ff978ff9700 1 -- 172.21.15.78:0/188048054 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff96c025460 con 0x7ff97c0fbcb0 2023-12-21T03:39:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.286+0000 7ff9824d8700 1 -- 172.21.15.78:0/188048054 --> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7ff97c0fda30 con 0x7ff95c02fdb0 2023-12-21T03:39:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.286+0000 7ff978ff9700 1 -- 172.21.15.78:0/188048054 <== mgr.14114 v2:172.21.15.78:6800/1548863549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7ff97c0fda30 con 0x7ff95c02fdb0 2023-12-21T03:39:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: value unchanged 2023-12-21T03:39:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.288+0000 7ff9824d8700 1 -- 172.21.15.78:0/188048054 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7ff95c02fdb0 msgr2=0x7ff95c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.288+0000 7ff9824d8700 1 --2- 172.21.15.78:0/188048054 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7ff95c02fdb0 0x7ff95c032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff9640096a0 tx=0x7ff964006b40).stop 2023-12-21T03:39:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.288+0000 7ff9824d8700 1 -- 172.21.15.78:0/188048054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 msgr2=0x7ff97c0725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:01.290 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.288+0000 7ff9824d8700 1 --2- 172.21.15.78:0/188048054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0725e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff96c00b1d0 tx=0x7ff96c006ee0).stop 2023-12-21T03:39:01.290 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.288+0000 7ff9824d8700 1 -- 172.21.15.78:0/188048054 shutdown_connections 2023-12-21T03:39:01.290 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.288+0000 7ff9824d8700 1 --2- 172.21.15.78:0/188048054 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7ff95c02fdb0 0x7ff95c032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:01.290 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.288+0000 7ff9824d8700 1 --2- 172.21.15.78:0/188048054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c0fbcb0 0x7ff97c0725e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:01.290 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.288+0000 7ff9824d8700 1 -- 172.21.15.78:0/188048054 >> 172.21.15.78:0/188048054 conn(0x7ff97c0f74b0 msgr2=0x7ff97c0f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:01.290 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.288+0000 7ff9824d8700 1 -- 172.21.15.78:0/188048054 shutdown_connections 2023-12-21T03:39:01.290 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:01.288+0000 7ff9824d8700 1 -- 172.21.15.78:0/188048054 wait complete. 2023-12-21T03:39:01.654 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:00.415589+0000 mgr.smithi078.zidbgq (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:01.654 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:00.440031+0000 mon.smithi078 (mon.0) 51 : cluster [DBG] mgrmap e8: smithi078.zidbgq(active, since 2s) 2023-12-21T03:39:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:Generating ssh key... 2023-12-21T03:39:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.001+0000 7f826aafa700 1 Processor -- start 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.001+0000 7f826aafa700 1 -- start start 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.001+0000 7f826aafa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f8264100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.001+0000 7f826aafa700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82640fc750 con 0x7f82640703b0 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.001+0000 7f8268896700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f8264100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.001+0000 7f8268896700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f8264100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39224/0 (socket says 172.21.15.78:39224) 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.001+0000 7f8268896700 1 -- 172.21.15.78:0/4161122343 learned_addr learned my addr 172.21.15.78:0/4161122343 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.001+0000 7f8268896700 1 -- 172.21.15.78:0/4161122343 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8264101250 con 0x7f82640703b0 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.001+0000 7f8268896700 1 --2- 172.21.15.78:0/4161122343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f8264100d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f825800b5f0 tx=0x7f8258006ee0).ready entity=mon.0 client_cookie=64cd6f77436dcd0a server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.002+0000 7f82637fe700 1 -- 172.21.15.78:0/4161122343 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f825800d600 con 0x7f82640703b0 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.002+0000 7f82637fe700 1 -- 172.21.15.78:0/4161122343 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f825800dbd0 con 0x7f82640703b0 2023-12-21T03:39:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.002+0000 7f82637fe700 1 -- 172.21.15.78:0/4161122343 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8258011c00 con 0x7f82640703b0 2023-12-21T03:39:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.002+0000 7f826aafa700 1 -- 172.21.15.78:0/4161122343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 msgr2=0x7f8264100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.002+0000 7f826aafa700 1 --2- 172.21.15.78:0/4161122343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f8264100d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f825800b5f0 tx=0x7f8258006ee0).stop 2023-12-21T03:39:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.002+0000 7f826aafa700 1 -- 172.21.15.78:0/4161122343 shutdown_connections 2023-12-21T03:39:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.002+0000 7f826aafa700 1 --2- 172.21.15.78:0/4161122343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f8264100d60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.002+0000 7f826aafa700 1 -- 172.21.15.78:0/4161122343 >> 172.21.15.78:0/4161122343 conn(0x7f82640f63c0 msgr2=0x7f82640f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.003+0000 7f826aafa700 1 -- 172.21.15.78:0/4161122343 shutdown_connections 2023-12-21T03:39:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.003+0000 7f826aafa700 1 -- 172.21.15.78:0/4161122343 wait complete. 2023-12-21T03:39:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.003+0000 7f826aafa700 1 Processor -- start 2023-12-21T03:39:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.003+0000 7f826aafa700 1 -- start start 2023-12-21T03:39:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f826aafa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f826410cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f826aafa700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826410d100 con 0x7f82640703b0 2023-12-21T03:39:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f8268896700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f826410cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f8268896700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f826410cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39230/0 (socket says 172.21.15.78:39230) 2023-12-21T03:39:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f8268896700 1 -- 172.21.15.78:0/3188566941 learned_addr learned my addr 172.21.15.78:0/3188566941 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f8268896700 1 -- 172.21.15.78:0/3188566941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825800b200 con 0x7f82640703b0 2023-12-21T03:39:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f8268896700 1 --2- 172.21.15.78:0/3188566941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f826410cc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f825800b190 tx=0x7f8258006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f82617fa700 1 -- 172.21.15.78:0/3188566941 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f825801b410 con 0x7f82640703b0 2023-12-21T03:39:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f82617fa700 1 -- 172.21.15.78:0/3188566941 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f825801b9e0 con 0x7f82640703b0 2023-12-21T03:39:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f82617fa700 1 -- 172.21.15.78:0/3188566941 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f825801a880 con 0x7f82640703b0 2023-12-21T03:39:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f826aafa700 1 -- 172.21.15.78:0/3188566941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826410d300 con 0x7f82640703b0 2023-12-21T03:39:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.004+0000 7f826aafa700 1 -- 172.21.15.78:0/3188566941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826410d710 con 0x7f82640703b0 2023-12-21T03:39:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.005+0000 7f82617fa700 1 -- 172.21.15.78:0/3188566941 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f825800f070 con 0x7f82640703b0 2023-12-21T03:39:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.005+0000 7f82617fa700 1 --2- 172.21.15.78:0/3188566941 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f824802fdb0 0x7f8248032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.005+0000 7f82617fa700 1 -- 172.21.15.78:0/3188566941 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f82580411f0 con 0x7f82640703b0 2023-12-21T03:39:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.005+0000 7f826aafa700 1 -- 172.21.15.78:0/3188566941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8264045ad0 con 0x7f82640703b0 2023-12-21T03:39:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.005+0000 7f8263fff700 1 --2- 172.21.15.78:0/3188566941 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f824802fdb0 0x7f8248032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.006+0000 7f8263fff700 1 --2- 172.21.15.78:0/3188566941 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f824802fdb0 0x7f8248032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f82500096a0 tx=0x7f8250006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.009+0000 7f82617fa700 1 -- 172.21.15.78:0/3188566941 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8258016070 con 0x7f82640703b0 2023-12-21T03:39:02.161 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.160+0000 7f826aafa700 1 -- 172.21.15.78:0/3188566941 --> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8264106b70 con 0x7f824802fdb0 2023-12-21T03:39:02.474 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.473+0000 7f82617fa700 1 -- 172.21.15.78:0/3188566941 <== mgr.14114 v2:172.21.15.78:6800/1548863549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8264106b70 con 0x7f824802fdb0 2023-12-21T03:39:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.475+0000 7f826aafa700 1 -- 172.21.15.78:0/3188566941 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f824802fdb0 msgr2=0x7f8248032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.475+0000 7f826aafa700 1 --2- 172.21.15.78:0/3188566941 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f824802fdb0 0x7f8248032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f82500096a0 tx=0x7f8250006b40).stop 2023-12-21T03:39:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.475+0000 7f826aafa700 1 -- 172.21.15.78:0/3188566941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 msgr2=0x7f826410cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.475+0000 7f826aafa700 1 --2- 172.21.15.78:0/3188566941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f826410cc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f825800b190 tx=0x7f8258006ee0).stop 2023-12-21T03:39:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.475+0000 7f826aafa700 1 -- 172.21.15.78:0/3188566941 shutdown_connections 2023-12-21T03:39:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.475+0000 7f826aafa700 1 --2- 172.21.15.78:0/3188566941 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f824802fdb0 0x7f8248032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.475+0000 7f826aafa700 1 --2- 172.21.15.78:0/3188566941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82640703b0 0x7f826410cc10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.475+0000 7f826aafa700 1 -- 172.21.15.78:0/3188566941 >> 172.21.15.78:0/3188566941 conn(0x7f82640f63c0 msgr2=0x7f82640f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.475+0000 7f826aafa700 1 -- 172.21.15.78:0/3188566941 shutdown_connections 2023-12-21T03:39:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:02.475+0000 7f826aafa700 1 -- 172.21.15.78:0/3188566941 wait complete. 2023-12-21T03:39:02.754 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:01.286811+0000 mgr.smithi078.zidbgq (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-12-21T03:39:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.179+0000 7f0ff477d700 1 Processor -- start 2023-12-21T03:39:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.179+0000 7f0ff477d700 1 -- start start 2023-12-21T03:39:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.179+0000 7f0ff477d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.180+0000 7f0ff477d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec0fedd0 con 0x7f0fec070430 2023-12-21T03:39:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.180+0000 7f0ff2519700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.180+0000 7f0ff2519700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39246/0 (socket says 172.21.15.78:39246) 2023-12-21T03:39:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.180+0000 7f0ff2519700 1 -- 172.21.15.78:0/2261822285 learned_addr learned my addr 172.21.15.78:0/2261822285 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:03.182 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.180+0000 7f0ff2519700 1 -- 172.21.15.78:0/2261822285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec0fef10 con 0x7f0fec070430 2023-12-21T03:39:03.182 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.180+0000 7f0ff2519700 1 --2- 172.21.15.78:0/2261822285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec0fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0fdc00b5f0 tx=0x7f0fdc006ee0).ready entity=mon.0 client_cookie=c8eed1954a4b32aa server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:03.183 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.180+0000 7f0ff1517700 1 -- 172.21.15.78:0/2261822285 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fdc00d600 con 0x7f0fec070430 2023-12-21T03:39:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.180+0000 7f0ff1517700 1 -- 172.21.15.78:0/2261822285 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0fdc00dbd0 con 0x7f0fec070430 2023-12-21T03:39:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.180+0000 7f0ff1517700 1 -- 172.21.15.78:0/2261822285 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fdc011c00 con 0x7f0fec070430 2023-12-21T03:39:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.181+0000 7f0ff477d700 1 -- 172.21.15.78:0/2261822285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 msgr2=0x7f0fec0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.181+0000 7f0ff477d700 1 --2- 172.21.15.78:0/2261822285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec0fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0fdc00b5f0 tx=0x7f0fdc006ee0).stop 2023-12-21T03:39:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.181+0000 7f0ff477d700 1 -- 172.21.15.78:0/2261822285 shutdown_connections 2023-12-21T03:39:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.181+0000 7f0ff477d700 1 --2- 172.21.15.78:0/2261822285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec0fe8e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.181+0000 7f0ff477d700 1 -- 172.21.15.78:0/2261822285 >> 172.21.15.78:0/2261822285 conn(0x7f0fec0f63c0 msgr2=0x7f0fec0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.181+0000 7f0ff477d700 1 -- 172.21.15.78:0/2261822285 shutdown_connections 2023-12-21T03:39:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.181+0000 7f0ff477d700 1 -- 172.21.15.78:0/2261822285 wait complete. 2023-12-21T03:39:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.182+0000 7f0ff477d700 1 Processor -- start 2023-12-21T03:39:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.182+0000 7f0ff477d700 1 -- start start 2023-12-21T03:39:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.182+0000 7f0ff477d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.182+0000 7f0ff477d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec10aef0 con 0x7f0fec070430 2023-12-21T03:39:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.182+0000 7f0ff2519700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.182+0000 7f0ff2519700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39258/0 (socket says 172.21.15.78:39258) 2023-12-21T03:39:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.182+0000 7f0ff2519700 1 -- 172.21.15.78:0/2682438643 learned_addr learned my addr 172.21.15.78:0/2682438643 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:03.186 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.182+0000 7f0ff2519700 1 -- 172.21.15.78:0/2682438643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc00b200 con 0x7f0fec070430 2023-12-21T03:39:03.186 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.183+0000 7f0ff2519700 1 --2- 172.21.15.78:0/2682438643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec10aa00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0fdc00b190 tx=0x7f0fdc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:03.186 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.183+0000 7f0fdaffd700 1 -- 172.21.15.78:0/2682438643 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fdc01b410 con 0x7f0fec070430 2023-12-21T03:39:03.186 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.183+0000 7f0fdaffd700 1 -- 172.21.15.78:0/2682438643 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0fdc01b9e0 con 0x7f0fec070430 2023-12-21T03:39:03.186 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.183+0000 7f0ff477d700 1 -- 172.21.15.78:0/2682438643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec10b0f0 con 0x7f0fec070430 2023-12-21T03:39:03.186 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.183+0000 7f0fdaffd700 1 -- 172.21.15.78:0/2682438643 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fdc01a880 con 0x7f0fec070430 2023-12-21T03:39:03.186 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.183+0000 7f0ff477d700 1 -- 172.21.15.78:0/2682438643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec10b500 con 0x7f0fec070430 2023-12-21T03:39:03.187 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.184+0000 7f0fdaffd700 1 -- 172.21.15.78:0/2682438643 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0fdc00f070 con 0x7f0fec070430 2023-12-21T03:39:03.187 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.184+0000 7f0fdaffd700 1 --2- 172.21.15.78:0/2682438643 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f0fd402fdb0 0x7f0fd4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:03.187 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.184+0000 7f0fdaffd700 1 -- 172.21.15.78:0/2682438643 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0fdc0411e0 con 0x7f0fec070430 2023-12-21T03:39:03.187 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.184+0000 7f0ff477d700 1 -- 172.21.15.78:0/2682438643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fec045ad0 con 0x7f0fec070430 2023-12-21T03:39:03.187 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.184+0000 7f0ff1d18700 1 --2- 172.21.15.78:0/2682438643 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f0fd402fdb0 0x7f0fd4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:03.187 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.184+0000 7f0ff1d18700 1 --2- 172.21.15.78:0/2682438643 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f0fd402fdb0 0x7f0fd4032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0fe80096a0 tx=0x7f0fe8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:03.189 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.188+0000 7f0fdaffd700 1 -- 172.21.15.78:0/2682438643 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fdc016070 con 0x7f0fec070430 2023-12-21T03:39:03.343 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.341+0000 7f0ff477d700 1 -- 172.21.15.78:0/2682438643 --> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fec10bde0 con 0x7f0fd402fdb0 2023-12-21T03:39:03.343 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.342+0000 7f0fdaffd700 1 -- 172.21.15.78:0/2682438643 <== mgr.14114 v2:172.21.15.78:6800/1548863549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f0fec10bde0 con 0x7f0fd402fdb0 2023-12-21T03:39:03.343 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4utIP2qTeLRtf9My0CWyNka/Jf0Vg/RAZiPO4lQi3qV/SH51SSFcPBoAAcg8OP/2mxyaEGlQ0Si1o5URr1eUghaUr3ZoPmokScOH6J/lml/1uVP7t2F4mwq5989q7XAWIQMe1PhNEkUZCkQQBt5wvSsP1tYMmBsod2r0CBhZmSooQY3GWt9npmh6HFmgWzK8W0UzwWB83SdTjZkoDhWtJbbkHjh0MntLbU+I17PuqU5yQKeruOafLHSWi/856DpEC1S3zGE9Vs1qfes9h2hy69/FahoYi93f/U5yKyYmL/IcOTw4hkfx5IYVDCsq8Q0B74AMtYpI9vpYOxMnz2Q813UMN/FW68F9VgIg7owGRN2QGZOoqAr5S+PSylj7JLpypzKaW0SXUoOg1Y4JFfqFWqDCDh+04kB8cgvmCNEuISI8HO/dAaIL/SIb9Z5HqgJpHg7XpP5Lp+2P4Yxp6K1Tu50EkcyYatJx1AubFhZ8Z6i/eikEzJlkoja+ny8hAp7M= ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:39:03.345 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.343+0000 7f0ff477d700 1 -- 172.21.15.78:0/2682438643 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f0fd402fdb0 msgr2=0x7f0fd4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:03.345 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.344+0000 7f0ff477d700 1 --2- 172.21.15.78:0/2682438643 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f0fd402fdb0 0x7f0fd4032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0fe80096a0 tx=0x7f0fe8006b40).stop 2023-12-21T03:39:03.345 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.344+0000 7f0ff477d700 1 -- 172.21.15.78:0/2682438643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 msgr2=0x7f0fec10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:03.345 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.344+0000 7f0ff477d700 1 --2- 172.21.15.78:0/2682438643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec10aa00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0fdc00b190 tx=0x7f0fdc006ee0).stop 2023-12-21T03:39:03.345 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.344+0000 7f0ff477d700 1 -- 172.21.15.78:0/2682438643 shutdown_connections 2023-12-21T03:39:03.345 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.344+0000 7f0ff477d700 1 --2- 172.21.15.78:0/2682438643 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f0fd402fdb0 0x7f0fd4032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:03.345 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.344+0000 7f0ff477d700 1 --2- 172.21.15.78:0/2682438643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec070430 0x7f0fec10aa00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:03.346 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.344+0000 7f0ff477d700 1 -- 172.21.15.78:0/2682438643 >> 172.21.15.78:0/2682438643 conn(0x7f0fec0f63c0 msgr2=0x7f0fec0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:03.346 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.344+0000 7f0ff477d700 1 -- 172.21.15.78:0/2682438643 shutdown_connections 2023-12-21T03:39:03.346 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:03.344+0000 7f0ff477d700 1 -- 172.21.15.78:0/2682438643 wait complete. 2023-12-21T03:39:03.710 INFO:teuthology.orchestra.run.smithi078.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-21T03:39:03.710 INFO:teuthology.orchestra.run.smithi078.stderr:Adding key to root@localhost authorized_keys... 2023-12-21T03:39:03.711 INFO:teuthology.orchestra.run.smithi078.stderr:Adding host smithi078... 2023-12-21T03:39:03.711 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi078 172.21.15.78 2023-12-21T03:39:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:02.161050+0000 mgr.smithi078.zidbgq (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2023-12-21T03:39:03.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:02.470213+0000 mon.smithi078 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:03.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:02.472651+0000 mon.smithi078 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:04.067 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.065+0000 7fdc2e31a700 1 Processor -- start 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.065+0000 7fdc2e31a700 1 -- start start 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.065+0000 7fdc2e31a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc28100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.065+0000 7fdc2e31a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc280fc770 con 0x7fdc28070390 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.065+0000 7fdc27fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc28100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.066+0000 7fdc27fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc28100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39260/0 (socket says 172.21.15.78:39260) 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.066+0000 7fdc27fff700 1 -- 172.21.15.78:0/1260165443 learned_addr learned my addr 172.21.15.78:0/1260165443 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.066+0000 7fdc27fff700 1 -- 172.21.15.78:0/1260165443 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc28101270 con 0x7fdc28070390 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.066+0000 7fdc27fff700 1 --2- 172.21.15.78:0/1260165443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc28100d80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fdc1c00b5f0 tx=0x7fdc1c006ee0).ready entity=mon.0 client_cookie=528603196e544d1e server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.066+0000 7fdc26ffd700 1 -- 172.21.15.78:0/1260165443 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc1c00d600 con 0x7fdc28070390 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.066+0000 7fdc26ffd700 1 -- 172.21.15.78:0/1260165443 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdc1c00dbd0 con 0x7fdc28070390 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.066+0000 7fdc26ffd700 1 -- 172.21.15.78:0/1260165443 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc1c011c00 con 0x7fdc28070390 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.067+0000 7fdc2e31a700 1 -- 172.21.15.78:0/1260165443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 msgr2=0x7fdc28100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:04.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.067+0000 7fdc2e31a700 1 --2- 172.21.15.78:0/1260165443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc28100d80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fdc1c00b5f0 tx=0x7fdc1c006ee0).stop 2023-12-21T03:39:04.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.067+0000 7fdc2e31a700 1 -- 172.21.15.78:0/1260165443 shutdown_connections 2023-12-21T03:39:04.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.067+0000 7fdc2e31a700 1 --2- 172.21.15.78:0/1260165443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc28100d80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:04.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.067+0000 7fdc2e31a700 1 -- 172.21.15.78:0/1260165443 >> 172.21.15.78:0/1260165443 conn(0x7fdc280f63c0 msgr2=0x7fdc280f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:04.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.067+0000 7fdc2e31a700 1 -- 172.21.15.78:0/1260165443 shutdown_connections 2023-12-21T03:39:04.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.067+0000 7fdc2e31a700 1 -- 172.21.15.78:0/1260165443 wait complete. 2023-12-21T03:39:04.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.068+0000 7fdc2e31a700 1 Processor -- start 2023-12-21T03:39:04.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.068+0000 7fdc2e31a700 1 -- start start 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.068+0000 7fdc2e31a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc2810cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.068+0000 7fdc2e31a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2810d0f0 con 0x7fdc28070390 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.068+0000 7fdc27fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc2810cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.068+0000 7fdc27fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc2810cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39264/0 (socket says 172.21.15.78:39264) 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.068+0000 7fdc27fff700 1 -- 172.21.15.78:0/2192599191 learned_addr learned my addr 172.21.15.78:0/2192599191 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.068+0000 7fdc27fff700 1 -- 172.21.15.78:0/2192599191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c00b200 con 0x7fdc28070390 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.068+0000 7fdc27fff700 1 --2- 172.21.15.78:0/2192599191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc2810cc00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fdc1c00b190 tx=0x7fdc1c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.069+0000 7fdc24ff9700 1 -- 172.21.15.78:0/2192599191 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc1c01b410 con 0x7fdc28070390 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.069+0000 7fdc24ff9700 1 -- 172.21.15.78:0/2192599191 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdc1c01b9e0 con 0x7fdc28070390 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.069+0000 7fdc2e31a700 1 -- 172.21.15.78:0/2192599191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2810d2f0 con 0x7fdc28070390 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.069+0000 7fdc24ff9700 1 -- 172.21.15.78:0/2192599191 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc1c01a880 con 0x7fdc28070390 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.069+0000 7fdc2e31a700 1 -- 172.21.15.78:0/2192599191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2810d700 con 0x7fdc28070390 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.069+0000 7fdc24ff9700 1 -- 172.21.15.78:0/2192599191 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fdc1c00f070 con 0x7fdc28070390 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.070+0000 7fdc24ff9700 1 --2- 172.21.15.78:0/2192599191 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7fdc1402fd50 0x7fdc14032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.070+0000 7fdc24ff9700 1 -- 172.21.15.78:0/2192599191 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fdc1c0413c0 con 0x7fdc28070390 2023-12-21T03:39:04.073 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.070+0000 7fdc277fe700 1 --2- 172.21.15.78:0/2192599191 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7fdc1402fd50 0x7fdc14032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:04.073 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.070+0000 7fdc277fe700 1 --2- 172.21.15.78:0/2192599191 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7fdc1402fd50 0x7fdc14032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdc180096a0 tx=0x7fdc18006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:04.073 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.070+0000 7fdc2e31a700 1 -- 172.21.15.78:0/2192599191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc28045ad0 con 0x7fdc28070390 2023-12-21T03:39:04.075 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.074+0000 7fdc24ff9700 1 -- 172.21.15.78:0/2192599191 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc1c010360 con 0x7fdc28070390 2023-12-21T03:39:04.249 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.248+0000 7fdc2e31a700 1 -- 172.21.15.78:0/2192599191 --> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc28106b00 con 0x7fdc1402fd50 2023-12-21T03:39:04.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:03.342370+0000 mgr.smithi078.zidbgq (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:04.914 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Added host 'smithi078' 2023-12-21T03:39:04.914 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.911+0000 7fdc24ff9700 1 -- 172.21.15.78:0/2192599191 <== mgr.14114 v2:172.21.15.78:6800/1548863549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fdc28106b00 con 0x7fdc1402fd50 2023-12-21T03:39:04.915 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.913+0000 7fdc2e31a700 1 -- 172.21.15.78:0/2192599191 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7fdc1402fd50 msgr2=0x7fdc14032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:04.916 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.913+0000 7fdc2e31a700 1 --2- 172.21.15.78:0/2192599191 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7fdc1402fd50 0x7fdc14032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdc180096a0 tx=0x7fdc18006b40).stop 2023-12-21T03:39:04.916 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.913+0000 7fdc2e31a700 1 -- 172.21.15.78:0/2192599191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 msgr2=0x7fdc2810cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:04.916 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.913+0000 7fdc2e31a700 1 --2- 172.21.15.78:0/2192599191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc2810cc00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fdc1c00b190 tx=0x7fdc1c006ee0).stop 2023-12-21T03:39:04.916 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.913+0000 7fdc2e31a700 1 -- 172.21.15.78:0/2192599191 shutdown_connections 2023-12-21T03:39:04.916 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.913+0000 7fdc2e31a700 1 --2- 172.21.15.78:0/2192599191 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7fdc1402fd50 0x7fdc14032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:04.916 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.913+0000 7fdc2e31a700 1 --2- 172.21.15.78:0/2192599191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc28070390 0x7fdc2810cc00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:04.916 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.913+0000 7fdc2e31a700 1 -- 172.21.15.78:0/2192599191 >> 172.21.15.78:0/2192599191 conn(0x7fdc280f63c0 msgr2=0x7fdc280f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:04.916 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.913+0000 7fdc2e31a700 1 -- 172.21.15.78:0/2192599191 shutdown_connections 2023-12-21T03:39:04.916 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:04.913+0000 7fdc2e31a700 1 -- 172.21.15.78:0/2192599191 wait complete. 2023-12-21T03:39:05.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:04.249222+0000 mgr.smithi078.zidbgq (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:05.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:04.911633+0000 mon.smithi078 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:05.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:04.912155+0000 mon.smithi078 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:39:06.468 INFO:teuthology.orchestra.run.smithi078.stderr:Deploying mon service with default placement... 2023-12-21T03:39:06.469 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-12-21T03:39:06.810 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:06 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:04.911924+0000 mgr.smithi078.zidbgq (mgr.14114) 9 : cephadm [INF] Added host smithi078 2023-12-21T03:39:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.539+0000 7f5a45f8b700 1 Processor -- start 2023-12-21T03:39:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.539+0000 7f5a45f8b700 1 -- start start 2023-12-21T03:39:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.539+0000 7f5a45f8b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.539+0000 7f5a45f8b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400fedd0 con 0x7f5a40070430 2023-12-21T03:39:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.540+0000 7f5a3f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.540+0000 7f5a3f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39266/0 (socket says 172.21.15.78:39266) 2023-12-21T03:39:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.540+0000 7f5a3f7fe700 1 -- 172.21.15.78:0/2132727833 learned_addr learned my addr 172.21.15.78:0/2132727833 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.540+0000 7f5a3f7fe700 1 -- 172.21.15.78:0/2132727833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400fef10 con 0x7f5a40070430 2023-12-21T03:39:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.540+0000 7f5a3f7fe700 1 --2- 172.21.15.78:0/2132727833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f5a3000b5f0 tx=0x7f5a30006ee0).ready entity=mon.0 client_cookie=83dbe1f7a235db6 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.540+0000 7f5a3e7fc700 1 -- 172.21.15.78:0/2132727833 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a3000d600 con 0x7f5a40070430 2023-12-21T03:39:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.540+0000 7f5a3e7fc700 1 -- 172.21.15.78:0/2132727833 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5a3000dbd0 con 0x7f5a40070430 2023-12-21T03:39:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.540+0000 7f5a3e7fc700 1 -- 172.21.15.78:0/2132727833 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a30011c00 con 0x7f5a40070430 2023-12-21T03:39:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.541+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2132727833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 msgr2=0x7f5a400fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.541+0000 7f5a45f8b700 1 --2- 172.21.15.78:0/2132727833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f5a3000b5f0 tx=0x7f5a30006ee0).stop 2023-12-21T03:39:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.541+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2132727833 shutdown_connections 2023-12-21T03:39:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.541+0000 7f5a45f8b700 1 --2- 172.21.15.78:0/2132727833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.541+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2132727833 >> 172.21.15.78:0/2132727833 conn(0x7f5a400f63c0 msgr2=0x7f5a400f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.541+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2132727833 shutdown_connections 2023-12-21T03:39:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.541+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2132727833 wait complete. 2023-12-21T03:39:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.541+0000 7f5a45f8b700 1 Processor -- start 2023-12-21T03:39:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.541+0000 7f5a45f8b700 1 -- start start 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a45f8b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a4010aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a45f8b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4010af30 con 0x7f5a40070430 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a3f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a4010aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a3f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a4010aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39268/0 (socket says 172.21.15.78:39268) 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a3f7fe700 1 -- 172.21.15.78:0/2451275387 learned_addr learned my addr 172.21.15.78:0/2451275387 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a3f7fe700 1 -- 172.21.15.78:0/2451275387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3000b200 con 0x7f5a40070430 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a3f7fe700 1 --2- 172.21.15.78:0/2451275387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a4010aa40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f5a3000e040 tx=0x7f5a30006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a27fff700 1 -- 172.21.15.78:0/2451275387 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a3000d9f0 con 0x7f5a40070430 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a27fff700 1 -- 172.21.15.78:0/2451275387 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5a3001a9f0 con 0x7f5a40070430 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a27fff700 1 -- 172.21.15.78:0/2451275387 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a3001b8c0 con 0x7f5a40070430 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2451275387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4010b130 con 0x7f5a40070430 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.542+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2451275387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4010b540 con 0x7f5a40070430 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.543+0000 7f5a27fff700 1 -- 172.21.15.78:0/2451275387 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5a3000f070 con 0x7f5a40070430 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.543+0000 7f5a27fff700 1 --2- 172.21.15.78:0/2451275387 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f5a2c02fdb0 0x7f5a2c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.543+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2451275387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a40045ad0 con 0x7f5a40070430 2023-12-21T03:39:07.631 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.543+0000 7f5a27fff700 1 -- 172.21.15.78:0/2451275387 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5a300412a0 con 0x7f5a40070430 2023-12-21T03:39:07.631 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.543+0000 7f5a3effd700 1 --2- 172.21.15.78:0/2451275387 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f5a2c02fdb0 0x7f5a2c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:07.631 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.544+0000 7f5a3effd700 1 --2- 172.21.15.78:0/2451275387 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f5a2c02fdb0 0x7f5a2c032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5a340096a0 tx=0x7f5a34006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:07.631 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.547+0000 7f5a27fff700 1 -- 172.21.15.78:0/2451275387 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a30022d20 con 0x7f5a40070430 2023-12-21T03:39:07.728 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.726+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2451275387 --> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a4010be20 con 0x7f5a2c02fdb0 2023-12-21T03:39:07.732 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.730+0000 7f5a27fff700 1 -- 172.21.15.78:0/2451275387 <== mgr.14114 v2:172.21.15.78:6800/1548863549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5a4010be20 con 0x7f5a2c02fdb0 2023-12-21T03:39:07.732 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Scheduled mon update... 2023-12-21T03:39:07.734 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.732+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2451275387 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f5a2c02fdb0 msgr2=0x7f5a2c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:07.734 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.732+0000 7f5a45f8b700 1 --2- 172.21.15.78:0/2451275387 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f5a2c02fdb0 0x7f5a2c032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5a340096a0 tx=0x7f5a34006b40).stop 2023-12-21T03:39:07.734 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.732+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2451275387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 msgr2=0x7f5a4010aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:07.734 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.732+0000 7f5a45f8b700 1 --2- 172.21.15.78:0/2451275387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a4010aa40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f5a3000e040 tx=0x7f5a30006ee0).stop 2023-12-21T03:39:07.734 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.732+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2451275387 shutdown_connections 2023-12-21T03:39:07.735 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.732+0000 7f5a45f8b700 1 --2- 172.21.15.78:0/2451275387 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f5a2c02fdb0 0x7f5a2c032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:07.735 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.732+0000 7f5a45f8b700 1 --2- 172.21.15.78:0/2451275387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a40070430 0x7f5a4010aa40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:07.735 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.732+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2451275387 >> 172.21.15.78:0/2451275387 conn(0x7f5a400f63c0 msgr2=0x7f5a400f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:07.735 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.732+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2451275387 shutdown_connections 2023-12-21T03:39:07.735 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:07.732+0000 7f5a45f8b700 1 -- 172.21.15.78:0/2451275387 wait complete. 2023-12-21T03:39:09.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:07.727992+0000 mgr.smithi078.zidbgq (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:09.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:07.728684+0000 mgr.smithi078.zidbgq (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-12-21T03:39:09.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:07.730976+0000 mon.smithi078 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:09.784 INFO:teuthology.orchestra.run.smithi078.stderr:Deploying mgr service with default placement... 2023-12-21T03:39:09.784 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-12-21T03:39:10.679 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.671+0000 7f1301788700 1 Processor -- start 2023-12-21T03:39:10.679 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.671+0000 7f1301788700 1 -- start start 2023-12-21T03:39:10.679 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.671+0000 7f1301788700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:10.680 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.671+0000 7f1301788700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc0fb2e0 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.680 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.671+0000 7f12fbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:10.680 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.672+0000 7f12fbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47158/0 (socket says 172.21.15.78:47158) 2023-12-21T03:39:10.680 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.672+0000 7f12fbfff700 1 -- 172.21.15.78:0/3983972533 learned_addr learned my addr 172.21.15.78:0/3983972533 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:10.680 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.672+0000 7f12fbfff700 1 -- 172.21.15.78:0/3983972533 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc0fb420 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.680 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.672+0000 7f12fbfff700 1 --2- 172.21.15.78:0/3983972533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc0fadf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f12ec00b5f0 tx=0x7f12ec006ee0).ready entity=mon.0 client_cookie=dbdcee1c9bb91d66 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:10.680 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.672+0000 7f12faffd700 1 -- 172.21.15.78:0/3983972533 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12ec00d600 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.680 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.672+0000 7f12faffd700 1 -- 172.21.15.78:0/3983972533 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f12ec00dbd0 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.680 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.672+0000 7f12faffd700 1 -- 172.21.15.78:0/3983972533 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12ec011c00 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.680 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.673+0000 7f1301788700 1 -- 172.21.15.78:0/3983972533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 msgr2=0x7f12fc0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.673+0000 7f1301788700 1 --2- 172.21.15.78:0/3983972533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc0fadf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f12ec00b5f0 tx=0x7f12ec006ee0).stop 2023-12-21T03:39:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.673+0000 7f1301788700 1 -- 172.21.15.78:0/3983972533 shutdown_connections 2023-12-21T03:39:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.673+0000 7f1301788700 1 --2- 172.21.15.78:0/3983972533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc0fadf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.673+0000 7f1301788700 1 -- 172.21.15.78:0/3983972533 >> 172.21.15.78:0/3983972533 conn(0x7f12fc0f6210 msgr2=0x7f12fc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.673+0000 7f1301788700 1 -- 172.21.15.78:0/3983972533 shutdown_connections 2023-12-21T03:39:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.673+0000 7f1301788700 1 -- 172.21.15.78:0/3983972533 wait complete. 2023-12-21T03:39:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.673+0000 7f1301788700 1 Processor -- start 2023-12-21T03:39:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.673+0000 7f1301788700 1 -- start start 2023-12-21T03:39:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f1301788700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc1085f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f1301788700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc108ae0 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f12fbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc1085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f12fbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc1085f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47160/0 (socket says 172.21.15.78:47160) 2023-12-21T03:39:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f12fbfff700 1 -- 172.21.15.78:0/2298310620 learned_addr learned my addr 172.21.15.78:0/2298310620 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f12fbfff700 1 -- 172.21.15.78:0/2298310620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ec00b200 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f12fbfff700 1 --2- 172.21.15.78:0/2298310620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc1085f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f12ec00b1d0 tx=0x7f12ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f12f8ff9700 1 -- 172.21.15.78:0/2298310620 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12ec01b410 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f12f8ff9700 1 -- 172.21.15.78:0/2298310620 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f12ec01b9e0 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f1301788700 1 -- 172.21.15.78:0/2298310620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12fc108ce0 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f12f8ff9700 1 -- 172.21.15.78:0/2298310620 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12ec01a840 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.674+0000 7f1301788700 1 -- 172.21.15.78:0/2298310620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12fc1090f0 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.675+0000 7f12f8ff9700 1 -- 172.21.15.78:0/2298310620 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f12ec00f070 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.675+0000 7f12f8ff9700 1 --2- 172.21.15.78:0/2298310620 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f12dc031fa0 0x7f12dc034460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.675+0000 7f12f8ff9700 1 -- 172.21.15.78:0/2298310620 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f12ec0412c0 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.675+0000 7f1301788700 1 -- 172.21.15.78:0/2298310620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12fc045ad0 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.675+0000 7f12fb7fe700 1 --2- 172.21.15.78:0/2298310620 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f12dc031fa0 0x7f12dc034460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.676+0000 7f12fb7fe700 1 --2- 172.21.15.78:0/2298310620 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f12dc031fa0 0x7f12dc034460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f12e40096a0 tx=0x7f12e4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.680+0000 7f12f8ff9700 1 -- 172.21.15.78:0/2298310620 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12ec016070 con 0x7f12fc0fa9d0 2023-12-21T03:39:10.858 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.856+0000 7f1301788700 1 -- 172.21.15.78:0/2298310620 --> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f12fc102500 con 0x7f12dc031fa0 2023-12-21T03:39:10.862 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.860+0000 7f12f8ff9700 1 -- 172.21.15.78:0/2298310620 <== mgr.14114 v2:172.21.15.78:6800/1548863549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f12fc102500 con 0x7f12dc031fa0 2023-12-21T03:39:10.862 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-12-21T03:39:10.864 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.862+0000 7f1301788700 1 -- 172.21.15.78:0/2298310620 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f12dc031fa0 msgr2=0x7f12dc034460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:10.864 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.862+0000 7f1301788700 1 --2- 172.21.15.78:0/2298310620 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f12dc031fa0 0x7f12dc034460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f12e40096a0 tx=0x7f12e4006b40).stop 2023-12-21T03:39:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.862+0000 7f1301788700 1 -- 172.21.15.78:0/2298310620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 msgr2=0x7f12fc1085f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.862+0000 7f1301788700 1 --2- 172.21.15.78:0/2298310620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc1085f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f12ec00b1d0 tx=0x7f12ec006ee0).stop 2023-12-21T03:39:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.862+0000 7f1301788700 1 -- 172.21.15.78:0/2298310620 shutdown_connections 2023-12-21T03:39:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.862+0000 7f1301788700 1 --2- 172.21.15.78:0/2298310620 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f12dc031fa0 0x7f12dc034460 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.862+0000 7f1301788700 1 --2- 172.21.15.78:0/2298310620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fa9d0 0x7f12fc1085f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.862+0000 7f1301788700 1 -- 172.21.15.78:0/2298310620 >> 172.21.15.78:0/2298310620 conn(0x7f12fc0f6210 msgr2=0x7f12fc0ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.862+0000 7f1301788700 1 -- 172.21.15.78:0/2298310620 shutdown_connections 2023-12-21T03:39:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:10.862+0000 7f1301788700 1 -- 172.21.15.78:0/2298310620 wait complete. 2023-12-21T03:39:11.250 INFO:teuthology.orchestra.run.smithi078.stderr:Deploying crash service with default placement... 2023-12-21T03:39:11.251 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-12-21T03:39:11.631 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.624+0000 7f03c952b700 1 Processor -- start 2023-12-21T03:39:11.631 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.625+0000 7f03c952b700 1 -- start start 2023-12-21T03:39:11.631 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.625+0000 7f03c952b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c4100d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:11.631 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.625+0000 7f03c952b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c40fc760 con 0x7f03c4070420 2023-12-21T03:39:11.631 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.625+0000 7f03c2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c4100d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:11.631 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.625+0000 7f03c2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c4100d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47168/0 (socket says 172.21.15.78:47168) 2023-12-21T03:39:11.631 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.625+0000 7f03c2ffd700 1 -- 172.21.15.78:0/3531227636 learned_addr learned my addr 172.21.15.78:0/3531227636 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:11.632 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.625+0000 7f03c2ffd700 1 -- 172.21.15.78:0/3531227636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c4101260 con 0x7f03c4070420 2023-12-21T03:39:11.632 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.625+0000 7f03c2ffd700 1 --2- 172.21.15.78:0/3531227636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c4100d70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f03b800b5f0 tx=0x7f03b8006ee0).ready entity=mon.0 client_cookie=f534f554fadd373a server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:11.632 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.625+0000 7f03c1ffb700 1 -- 172.21.15.78:0/3531227636 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03b800d600 con 0x7f03c4070420 2023-12-21T03:39:11.632 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.625+0000 7f03c1ffb700 1 -- 172.21.15.78:0/3531227636 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f03b800dbd0 con 0x7f03c4070420 2023-12-21T03:39:11.632 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.625+0000 7f03c1ffb700 1 -- 172.21.15.78:0/3531227636 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03b8011c00 con 0x7f03c4070420 2023-12-21T03:39:11.632 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.626+0000 7f03c952b700 1 -- 172.21.15.78:0/3531227636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 msgr2=0x7f03c4100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:11.633 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.626+0000 7f03c952b700 1 --2- 172.21.15.78:0/3531227636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c4100d70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f03b800b5f0 tx=0x7f03b8006ee0).stop 2023-12-21T03:39:11.633 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.626+0000 7f03c952b700 1 -- 172.21.15.78:0/3531227636 shutdown_connections 2023-12-21T03:39:11.633 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.626+0000 7f03c952b700 1 --2- 172.21.15.78:0/3531227636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c4100d70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:11.633 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.626+0000 7f03c952b700 1 -- 172.21.15.78:0/3531227636 >> 172.21.15.78:0/3531227636 conn(0x7f03c40f63b0 msgr2=0x7f03c40f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:11.633 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.626+0000 7f03c952b700 1 -- 172.21.15.78:0/3531227636 shutdown_connections 2023-12-21T03:39:11.633 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.626+0000 7f03c952b700 1 -- 172.21.15.78:0/3531227636 wait complete. 2023-12-21T03:39:11.633 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.627+0000 7f03c952b700 1 Processor -- start 2023-12-21T03:39:11.633 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.627+0000 7f03c952b700 1 -- start start 2023-12-21T03:39:11.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.627+0000 7f03c952b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c410ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:11.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.627+0000 7f03c952b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c410d1c0 con 0x7f03c4070420 2023-12-21T03:39:11.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.627+0000 7f03c2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c410ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:11.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.627+0000 7f03c2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c410ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47178/0 (socket says 172.21.15.78:47178) 2023-12-21T03:39:11.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.627+0000 7f03c2ffd700 1 -- 172.21.15.78:0/1236637940 learned_addr learned my addr 172.21.15.78:0/1236637940 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:11.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.628+0000 7f03c2ffd700 1 -- 172.21.15.78:0/1236637940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b800b200 con 0x7f03c4070420 2023-12-21T03:39:11.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.628+0000 7f03c2ffd700 1 --2- 172.21.15.78:0/1236637940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c410ccd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f03b800e040 tx=0x7f03b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:11.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.628+0000 7f03ab7fe700 1 -- 172.21.15.78:0/1236637940 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03b800d9f0 con 0x7f03c4070420 2023-12-21T03:39:11.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.628+0000 7f03ab7fe700 1 -- 172.21.15.78:0/1236637940 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f03b801a9f0 con 0x7f03c4070420 2023-12-21T03:39:11.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.628+0000 7f03ab7fe700 1 -- 172.21.15.78:0/1236637940 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03b801b8c0 con 0x7f03c4070420 2023-12-21T03:39:11.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.628+0000 7f03c952b700 1 -- 172.21.15.78:0/1236637940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c410d3c0 con 0x7f03c4070420 2023-12-21T03:39:11.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.628+0000 7f03c952b700 1 -- 172.21.15.78:0/1236637940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c410d7d0 con 0x7f03c4070420 2023-12-21T03:39:11.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.629+0000 7f03ab7fe700 1 -- 172.21.15.78:0/1236637940 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f03b800f070 con 0x7f03c4070420 2023-12-21T03:39:11.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.629+0000 7f03ab7fe700 1 --2- 172.21.15.78:0/1236637940 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f03b002fd50 0x7f03b0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:11.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.629+0000 7f03ab7fe700 1 -- 172.21.15.78:0/1236637940 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f03b8041310 con 0x7f03c4070420 2023-12-21T03:39:11.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.629+0000 7f03c27fc700 1 --2- 172.21.15.78:0/1236637940 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f03b002fd50 0x7f03b0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:11.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.629+0000 7f03c952b700 1 -- 172.21.15.78:0/1236637940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c4045ad0 con 0x7f03c4070420 2023-12-21T03:39:11.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.629+0000 7f03c27fc700 1 --2- 172.21.15.78:0/1236637940 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f03b002fd50 0x7f03b0032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f03b40096a0 tx=0x7f03b4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:11.636 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.633+0000 7f03ab7fe700 1 -- 172.21.15.78:0/1236637940 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03b8016070 con 0x7f03c4070420 2023-12-21T03:39:11.810 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.808+0000 7f03c952b700 1 -- 172.21.15.78:0/1236637940 --> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f03c410ecc0 con 0x7f03b002fd50 2023-12-21T03:39:11.814 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.812+0000 7f03ab7fe700 1 -- 172.21.15.78:0/1236637940 <== mgr.14114 v2:172.21.15.78:6800/1548863549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f03c410ecc0 con 0x7f03b002fd50 2023-12-21T03:39:11.814 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Scheduled crash update... 2023-12-21T03:39:11.816 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.814+0000 7f03c952b700 1 -- 172.21.15.78:0/1236637940 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f03b002fd50 msgr2=0x7f03b0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:11.816 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.814+0000 7f03c952b700 1 --2- 172.21.15.78:0/1236637940 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f03b002fd50 0x7f03b0032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f03b40096a0 tx=0x7f03b4006b40).stop 2023-12-21T03:39:11.816 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.814+0000 7f03c952b700 1 -- 172.21.15.78:0/1236637940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 msgr2=0x7f03c410ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:11.817 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.814+0000 7f03c952b700 1 --2- 172.21.15.78:0/1236637940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c410ccd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f03b800e040 tx=0x7f03b8006ee0).stop 2023-12-21T03:39:11.817 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.814+0000 7f03c952b700 1 -- 172.21.15.78:0/1236637940 shutdown_connections 2023-12-21T03:39:11.817 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.814+0000 7f03c952b700 1 --2- 172.21.15.78:0/1236637940 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f03b002fd50 0x7f03b0032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:11.817 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.814+0000 7f03c952b700 1 --2- 172.21.15.78:0/1236637940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c4070420 0x7f03c410ccd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:11.817 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.814+0000 7f03c952b700 1 -- 172.21.15.78:0/1236637940 >> 172.21.15.78:0/1236637940 conn(0x7f03c40f63b0 msgr2=0x7f03c40f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:11.817 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.815+0000 7f03c952b700 1 -- 172.21.15.78:0/1236637940 shutdown_connections 2023-12-21T03:39:11.817 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:11.815+0000 7f03c952b700 1 -- 172.21.15.78:0/1236637940 wait complete. 2023-12-21T03:39:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:Enabling mgr prometheus module... 2023-12-21T03:39:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-12-21T03:39:12.252 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:10.858277+0000 mgr.smithi078.zidbgq (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:12.252 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:10.858848+0000 mgr.smithi078.zidbgq (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-12-21T03:39:12.252 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:10.861163+0000 mon.smithi078 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:12.252 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:11.813525+0000 mon.smithi078 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.677+0000 7f8bfaca8700 1 Processor -- start 2023-12-21T03:39:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.677+0000 7f8bfaca8700 1 -- start start 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.678+0000 7f8bfaca8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf40fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.678+0000 7f8bfaca8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40fc5c0 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.678+0000 7f8bf8a44700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf40fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.678+0000 7f8bf8a44700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf40fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47182/0 (socket says 172.21.15.78:47182) 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.678+0000 7f8bf8a44700 1 -- 172.21.15.78:0/1190853216 learned_addr learned my addr 172.21.15.78:0/1190853216 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.678+0000 7f8bf8a44700 1 -- 172.21.15.78:0/1190853216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf40fc700 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.678+0000 7f8bf8a44700 1 --2- 172.21.15.78:0/1190853216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf40fc0d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f8be800b5f0 tx=0x7f8be8006ee0).ready entity=mon.0 client_cookie=4c2c8e2c51a33319 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.678+0000 7f8bf37fe700 1 -- 172.21.15.78:0/1190853216 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8be800d600 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.678+0000 7f8bf37fe700 1 -- 172.21.15.78:0/1190853216 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8be800dbd0 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.678+0000 7f8bf37fe700 1 -- 172.21.15.78:0/1190853216 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8be8011c00 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.679+0000 7f8bfaca8700 1 -- 172.21.15.78:0/1190853216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 msgr2=0x7f8bf40fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.679+0000 7f8bfaca8700 1 --2- 172.21.15.78:0/1190853216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf40fc0d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f8be800b5f0 tx=0x7f8be8006ee0).stop 2023-12-21T03:39:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.679+0000 7f8bfaca8700 1 -- 172.21.15.78:0/1190853216 shutdown_connections 2023-12-21T03:39:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.679+0000 7f8bfaca8700 1 --2- 172.21.15.78:0/1190853216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf40fc0d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.679+0000 7f8bfaca8700 1 -- 172.21.15.78:0/1190853216 >> 172.21.15.78:0/1190853216 conn(0x7f8bf40f74b0 msgr2=0x7f8bf40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.679+0000 7f8bfaca8700 1 -- 172.21.15.78:0/1190853216 shutdown_connections 2023-12-21T03:39:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.679+0000 7f8bfaca8700 1 -- 172.21.15.78:0/1190853216 wait complete. 2023-12-21T03:39:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.680+0000 7f8bfaca8700 1 Processor -- start 2023-12-21T03:39:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.680+0000 7f8bfaca8700 1 -- start start 2023-12-21T03:39:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.680+0000 7f8bfaca8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf410aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.680+0000 7f8bfaca8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf410af30 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.680+0000 7f8bf8a44700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf410aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.680+0000 7f8bf8a44700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf410aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47198/0 (socket says 172.21.15.78:47198) 2023-12-21T03:39:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.680+0000 7f8bf8a44700 1 -- 172.21.15.78:0/3983826581 learned_addr learned my addr 172.21.15.78:0/3983826581 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:12.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.680+0000 7f8bf8a44700 1 -- 172.21.15.78:0/3983826581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be800b200 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.680+0000 7f8bf8a44700 1 --2- 172.21.15.78:0/3983826581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf410aa40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f8be800e040 tx=0x7f8be8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:12.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.681+0000 7f8bf17fa700 1 -- 172.21.15.78:0/3983826581 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8be801b410 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.681+0000 7f8bf17fa700 1 -- 172.21.15.78:0/3983826581 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8be801b9e0 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.681+0000 7f8bf17fa700 1 -- 172.21.15.78:0/3983826581 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8be801a820 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.685 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.681+0000 7f8bfaca8700 1 -- 172.21.15.78:0/3983826581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf410b130 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.681+0000 7f8bfaca8700 1 -- 172.21.15.78:0/3983826581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf410b540 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.682+0000 7f8bf17fa700 1 -- 172.21.15.78:0/3983826581 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f8be800f070 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.682+0000 7f8bfaca8700 1 -- 172.21.15.78:0/3983826581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf4045ad0 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.682+0000 7f8bf17fa700 1 --2- 172.21.15.78:0/3983826581 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f8bd802fdb0 0x7f8bd8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:12.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.682+0000 7f8bf17fa700 1 -- 172.21.15.78:0/3983826581 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8be8041390 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.682+0000 7f8bf3fff700 1 --2- 172.21.15.78:0/3983826581 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f8bd802fdb0 0x7f8bd8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:12.686 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.682+0000 7f8bf3fff700 1 --2- 172.21.15.78:0/3983826581 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f8bd802fdb0 0x7f8bd8032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8be00096a0 tx=0x7f8be0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:12.687 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.686+0000 7f8bf17fa700 1 -- 172.21.15.78:0/3983826581 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8be8022310 con 0x7f8bf40fbcb0 2023-12-21T03:39:12.880 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:12.879+0000 7f8bfaca8700 1 -- 172.21.15.78:0/3983826581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f8bf40fd9c0 con 0x7f8bf40fbcb0 2023-12-21T03:39:13.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:11.810523+0000 mgr.smithi078.zidbgq (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:13.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:11.811193+0000 mgr.smithi078.zidbgq (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2023-12-21T03:39:13.154 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.152+0000 7f8bf17fa700 1 -- 172.21.15.78:0/3983826581 <== mon.0 v2:172.21.15.78:3300/0 7 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8be8022310 con 0x7f8bf40fbcb0 2023-12-21T03:39:13.865 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.863+0000 7f8bf17fa700 1 -- 172.21.15.78:0/3983826581 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f8be801a980 con 0x7f8bf40fbcb0 2023-12-21T03:39:13.866 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.864+0000 7f8bf17fa700 1 -- 172.21.15.78:0/3983826581 <== mon.0 v2:172.21.15.78:3300/0 9 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f8be8010360 con 0x7f8bf40fbcb0 2023-12-21T03:39:13.868 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.866+0000 7f8bfaca8700 1 -- 172.21.15.78:0/3983826581 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f8bd802fdb0 msgr2=0x7f8bd8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:13.868 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.866+0000 7f8bfaca8700 1 --2- 172.21.15.78:0/3983826581 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f8bd802fdb0 0x7f8bd8032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8be00096a0 tx=0x7f8be0006b40).stop 2023-12-21T03:39:13.869 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.866+0000 7f8bfaca8700 1 -- 172.21.15.78:0/3983826581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 msgr2=0x7f8bf410aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:13.869 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.866+0000 7f8bfaca8700 1 --2- 172.21.15.78:0/3983826581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf410aa40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f8be800e040 tx=0x7f8be8006ee0).stop 2023-12-21T03:39:13.869 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.866+0000 7f8bfaca8700 1 -- 172.21.15.78:0/3983826581 shutdown_connections 2023-12-21T03:39:13.869 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.866+0000 7f8bfaca8700 1 --2- 172.21.15.78:0/3983826581 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f8bd802fdb0 0x7f8bd8032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:13.869 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.866+0000 7f8bfaca8700 1 --2- 172.21.15.78:0/3983826581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bf40fbcb0 0x7f8bf410aa40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:13.869 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.866+0000 7f8bfaca8700 1 -- 172.21.15.78:0/3983826581 >> 172.21.15.78:0/3983826581 conn(0x7f8bf40f74b0 msgr2=0x7f8bf40f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:13.869 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.866+0000 7f8bfaca8700 1 -- 172.21.15.78:0/3983826581 shutdown_connections 2023-12-21T03:39:13.869 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:13.866+0000 7f8bfaca8700 1 -- 172.21.15.78:0/3983826581 wait complete. 2023-12-21T03:39:14.166 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:12.880420+0000 mon.smithi078 (mon.0) 59 : audit [INF] from='client.? 172.21.15.78:0/3983826581' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-21T03:39:14.166 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:13.153882+0000 mon.smithi078 (mon.0) 60 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:14.166 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:13.365644+0000 mon.smithi078 (mon.0) 61 : audit [INF] from='mgr.14114 172.21.15.78:0/1738120404' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:14.350 INFO:teuthology.orchestra.run.smithi078.stderr:Deploying prometheus service with default placement... 2023-12-21T03:39:14.351 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-12-21T03:39:14.742 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.733+0000 7f26ea475700 1 Processor -- start 2023-12-21T03:39:14.742 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.733+0000 7f26ea475700 1 -- start start 2023-12-21T03:39:14.742 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.733+0000 7f26ea475700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e4100b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:14.742 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.733+0000 7f26ea475700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e40fc520 con 0x7f26e4070430 2023-12-21T03:39:14.742 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.733+0000 7f26e9473700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e4100b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:14.742 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.733+0000 7f26e9473700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e4100b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47234/0 (socket says 172.21.15.78:47234) 2023-12-21T03:39:14.743 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.733+0000 7f26e9473700 1 -- 172.21.15.78:0/3565665336 learned_addr learned my addr 172.21.15.78:0/3565665336 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:14.743 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.733+0000 7f26e9473700 1 -- 172.21.15.78:0/3565665336 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e4101020 con 0x7f26e4070430 2023-12-21T03:39:14.743 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.733+0000 7f26e9473700 1 --2- 172.21.15.78:0/3565665336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e4100b30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f26e000b5f0 tx=0x7f26e0006ee0).ready entity=mon.0 client_cookie=f3b3adc063dd8742 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:14.743 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.734+0000 7f26d7fff700 1 -- 172.21.15.78:0/3565665336 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e000d600 con 0x7f26e4070430 2023-12-21T03:39:14.743 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.734+0000 7f26d7fff700 1 -- 172.21.15.78:0/3565665336 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26e000dc10 con 0x7f26e4070430 2023-12-21T03:39:14.743 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.734+0000 7f26d7fff700 1 -- 172.21.15.78:0/3565665336 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e0011cd0 con 0x7f26e4070430 2023-12-21T03:39:14.743 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.734+0000 7f26ea475700 1 -- 172.21.15.78:0/3565665336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 msgr2=0x7f26e4100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:14.743 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.734+0000 7f26ea475700 1 --2- 172.21.15.78:0/3565665336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e4100b30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f26e000b5f0 tx=0x7f26e0006ee0).stop 2023-12-21T03:39:14.744 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.734+0000 7f26ea475700 1 -- 172.21.15.78:0/3565665336 shutdown_connections 2023-12-21T03:39:14.744 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.734+0000 7f26ea475700 1 --2- 172.21.15.78:0/3565665336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e4100b30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:14.744 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.734+0000 7f26ea475700 1 -- 172.21.15.78:0/3565665336 >> 172.21.15.78:0/3565665336 conn(0x7f26e40f6210 msgr2=0x7f26e40f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:14.744 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.735+0000 7f26ea475700 1 -- 172.21.15.78:0/3565665336 shutdown_connections 2023-12-21T03:39:14.744 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.735+0000 7f26ea475700 1 -- 172.21.15.78:0/3565665336 wait complete. 2023-12-21T03:39:14.744 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.735+0000 7f26ea475700 1 Processor -- start 2023-12-21T03:39:14.744 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.735+0000 7f26ea475700 1 -- start start 2023-12-21T03:39:14.744 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26ea475700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e410caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:14.745 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26ea475700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e410cfe0 con 0x7f26e4070430 2023-12-21T03:39:14.745 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26e9473700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e410caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:14.745 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26e9473700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e410caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47244/0 (socket says 172.21.15.78:47244) 2023-12-21T03:39:14.745 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26e9473700 1 -- 172.21.15.78:0/1629304767 learned_addr learned my addr 172.21.15.78:0/1629304767 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:14.745 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26e9473700 1 -- 172.21.15.78:0/1629304767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e000b200 con 0x7f26e4070430 2023-12-21T03:39:14.745 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26e9473700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e410caf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f26e000df90 tx=0x7f26e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:14.745 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26d5ffb700 1 -- 172.21.15.78:0/1629304767 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e0011da0 con 0x7f26e4070430 2023-12-21T03:39:14.745 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26d5ffb700 1 -- 172.21.15.78:0/1629304767 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26e001b8c0 con 0x7f26e4070430 2023-12-21T03:39:14.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26ea475700 1 -- 172.21.15.78:0/1629304767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e410d1e0 con 0x7f26e4070430 2023-12-21T03:39:14.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26d5ffb700 1 -- 172.21.15.78:0/1629304767 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e001a930 con 0x7f26e4070430 2023-12-21T03:39:14.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.736+0000 7f26ea475700 1 -- 172.21.15.78:0/1629304767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e410d5f0 con 0x7f26e4070430 2023-12-21T03:39:14.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.737+0000 7f26d5ffb700 1 -- 172.21.15.78:0/1629304767 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f26e000f070 con 0x7f26e4070430 2023-12-21T03:39:14.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.737+0000 7f26d5ffb700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 0x7f26cc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:14.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.737+0000 7f26d5ffb700 1 -- 172.21.15.78:0/1629304767 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f26e00468e0 con 0x7f26e4070430 2023-12-21T03:39:14.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.737+0000 7f26e8c72700 1 -- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 msgr2=0x7f26cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1548863549 2023-12-21T03:39:14.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.737+0000 7f26e8c72700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 0x7f26cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-21T03:39:14.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.738+0000 7f26ea475700 1 -- 172.21.15.78:0/1629304767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e4045ad0 con 0x7f26e4070430 2023-12-21T03:39:14.747 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.742+0000 7f26d5ffb700 1 -- 172.21.15.78:0/1629304767 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26e0016070 con 0x7f26e4070430 2023-12-21T03:39:14.925 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.923+0000 7f26ea475700 1 -- 172.21.15.78:0/1629304767 --> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f26e4106a20 con 0x7f26cc02fdb0 2023-12-21T03:39:14.939 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.938+0000 7f26e8c72700 1 -- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 msgr2=0x7f26cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1548863549 2023-12-21T03:39:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:14.938+0000 7f26e8c72700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 0x7f26cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-21T03:39:15.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:13.865789+0000 mon.smithi078 (mon.0) 62 : audit [INF] from='client.? 172.21.15.78:0/3983826581' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-21T03:39:15.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:13.865857+0000 mon.smithi078 (mon.0) 63 : cluster [DBG] mgrmap e9: smithi078.zidbgq(active, since 15s) 2023-12-21T03:39:15.341 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:15.339+0000 7f26e8c72700 1 -- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 msgr2=0x7f26cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1548863549 2023-12-21T03:39:15.341 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:15.339+0000 7f26e8c72700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 0x7f26cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-21T03:39:16.142 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:16.140+0000 7f26e8c72700 1 -- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 msgr2=0x7f26cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1548863549 2023-12-21T03:39:16.142 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:16.140+0000 7f26e8c72700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 0x7f26cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-21T03:39:17.743 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:17.741+0000 7f26e8c72700 1 -- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 msgr2=0x7f26cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1548863549 2023-12-21T03:39:17.743 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:17.741+0000 7f26e8c72700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 0x7f26cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-21T03:39:18.816 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:18.814+0000 7f26d5ffb700 1 -- 172.21.15.78:0/1629304767 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f26e0024930 con 0x7f26e4070430 2023-12-21T03:39:18.816 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:18.814+0000 7f26d5ffb700 1 -- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 msgr2=0x7f26cc032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-21T03:39:18.816 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:18.814+0000 7f26d5ffb700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 0x7f26cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:39:19.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:18 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:18.762949+0000 mon.smithi078 (mon.0) 64 : cluster [INF] Active manager daemon smithi078.zidbgq restarted 2023-12-21T03:39:19.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:18 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:18.763331+0000 mon.smithi078 (mon.0) 65 : cluster [INF] Activating manager daemon smithi078.zidbgq 2023-12-21T03:39:19.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:18 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:18.765490+0000 mon.smithi078 (mon.0) 66 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-12-21T03:39:19.820 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.818+0000 7f26d5ffb700 1 -- 172.21.15.78:0/1629304767 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f26e001b5e0 con 0x7f26e4070430 2023-12-21T03:39:19.820 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.818+0000 7f26d5ffb700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f26cc033c20 0x7f26cc036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:19.820 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.818+0000 7f26d5ffb700 1 -- 172.21.15.78:0/1629304767 --> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f26e4106a20 con 0x7f26cc033c20 2023-12-21T03:39:19.821 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.819+0000 7f26e8c72700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f26cc033c20 0x7f26cc036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:19.821 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.819+0000 7f26e8c72700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f26cc033c20 0x7f26cc036010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f26d80036d0 tx=0x7f26d8006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.824+0000 7f26d5ffb700 1 -- 172.21.15.78:0/1629304767 <== mgr.14142 v2:172.21.15.78:6800/1046721827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f26e4106a20 con 0x7f26cc033c20 2023-12-21T03:39:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-12-21T03:39:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.826+0000 7f26ea475700 1 -- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f26cc033c20 msgr2=0x7f26cc036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.826+0000 7f26ea475700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f26cc033c20 0x7f26cc036010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f26d80036d0 tx=0x7f26d8006cb0).stop 2023-12-21T03:39:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.826+0000 7f26ea475700 1 -- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 msgr2=0x7f26e410caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.826+0000 7f26ea475700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e410caf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f26e000df90 tx=0x7f26e0006ee0).stop 2023-12-21T03:39:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.827+0000 7f26ea475700 1 -- 172.21.15.78:0/1629304767 shutdown_connections 2023-12-21T03:39:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.827+0000 7f26ea475700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f26cc033c20 0x7f26cc036010 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.827+0000 7f26ea475700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e4070430 0x7f26e410caf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.827+0000 7f26ea475700 1 --2- 172.21.15.78:0/1629304767 >> [v2:172.21.15.78:6800/1548863549,v1:172.21.15.78:6801/1548863549] conn(0x7f26cc02fdb0 0x7f26cc032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:39:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.827+0000 7f26ea475700 1 -- 172.21.15.78:0/1629304767 >> 172.21.15.78:0/1629304767 conn(0x7f26e40f6210 msgr2=0x7f26e40f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.828+0000 7f26ea475700 1 -- 172.21.15.78:0/1629304767 shutdown_connections 2023-12-21T03:39:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:19.828+0000 7f26ea475700 1 -- 172.21.15.78:0/1629304767 wait complete. 2023-12-21T03:39:20.016 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:18.816357+0000 mon.smithi078 (mon.0) 67 : cluster [DBG] mgrmap e10: smithi078.zidbgq(active, starting, since 0.0531139s) 2023-12-21T03:39:20.016 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:18.819426+0000 mon.smithi078 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2023-12-21T03:39:20.016 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:18.819654+0000 mon.smithi078 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mgr metadata", "who": "smithi078.zidbgq", "id": "smithi078.zidbgq"}]: dispatch 2023-12-21T03:39:20.016 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:18.819831+0000 mon.smithi078 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-21T03:39:20.016 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:18.819973+0000 mon.smithi078 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-21T03:39:20.016 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:18.820116+0000 mon.smithi078 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-21T03:39:20.016 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:18.833574+0000 mon.smithi078 (mon.0) 73 : cluster [INF] Manager daemon smithi078.zidbgq is now available 2023-12-21T03:39:20.016 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:18.931169+0000 mon.smithi078 (mon.0) 74 : audit [DBG] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:39:20.016 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:18.932632+0000 mon.smithi078 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:39:20.017 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:18.948010+0000 mon.smithi078 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:39:20.017 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:18.956351+0000 mon.smithi078 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:39:20.017 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:19.769455+0000 mon.smithi078 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:20.191 INFO:teuthology.orchestra.run.smithi078.stderr:Deploying grafana service with default placement... 2023-12-21T03:39:20.191 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-12-21T03:39:20.703 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.700+0000 7f49b4348700 1 Processor -- start 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.700+0000 7f49b4348700 1 -- start start 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.701+0000 7f49b4348700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac070a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.701+0000 7f49b4348700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac072900 con 0x7f49ac0724e0 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.701+0000 7f49b20e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.701+0000 7f49b20e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac070a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38692/0 (socket says 172.21.15.78:38692) 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.701+0000 7f49b20e4700 1 -- 172.21.15.78:0/2206594138 learned_addr learned my addr 172.21.15.78:0/2206594138 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.701+0000 7f49b20e4700 1 -- 172.21.15.78:0/2206594138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac070f70 con 0x7f49ac0724e0 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.701+0000 7f49b20e4700 1 --2- 172.21.15.78:0/2206594138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac070a80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f499c009960 tx=0x7f499c0092d0).ready entity=mon.0 client_cookie=82bc58f0abf718aa server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.702+0000 7f49b10e2700 1 -- 172.21.15.78:0/2206594138 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f499c00b9c0 con 0x7f49ac0724e0 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.702+0000 7f49b10e2700 1 -- 172.21.15.78:0/2206594138 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f499c011400 con 0x7f49ac0724e0 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.702+0000 7f49b10e2700 1 -- 172.21.15.78:0/2206594138 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f499c010580 con 0x7f49ac0724e0 2023-12-21T03:39:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.702+0000 7f49b4348700 1 -- 172.21.15.78:0/2206594138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 msgr2=0x7f49ac070a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.702+0000 7f49b4348700 1 --2- 172.21.15.78:0/2206594138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac070a80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f499c009960 tx=0x7f499c0092d0).stop 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.702+0000 7f49b4348700 1 -- 172.21.15.78:0/2206594138 shutdown_connections 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.702+0000 7f49b4348700 1 --2- 172.21.15.78:0/2206594138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac070a80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.702+0000 7f49b4348700 1 -- 172.21.15.78:0/2206594138 >> 172.21.15.78:0/2206594138 conn(0x7f49ac0f74b0 msgr2=0x7f49ac0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.702+0000 7f49b4348700 1 -- 172.21.15.78:0/2206594138 shutdown_connections 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.702+0000 7f49b4348700 1 -- 172.21.15.78:0/2206594138 wait complete. 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.703+0000 7f49b4348700 1 Processor -- start 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.703+0000 7f49b4348700 1 -- start start 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.703+0000 7f49b4348700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac10cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.703+0000 7f49b4348700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac10d200 con 0x7f49ac0724e0 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.703+0000 7f49b20e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac10cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.703+0000 7f49b20e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac10cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38708/0 (socket says 172.21.15.78:38708) 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.703+0000 7f49b20e4700 1 -- 172.21.15.78:0/1708416593 learned_addr learned my addr 172.21.15.78:0/1708416593 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.703+0000 7f49b20e4700 1 -- 172.21.15.78:0/1708416593 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499c009570 con 0x7f49ac0724e0 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.704+0000 7f49b20e4700 1 --2- 172.21.15.78:0/1708416593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac10cd10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f499c011cf0 tx=0x7f499c0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.704+0000 7f49aaffd700 1 -- 172.21.15.78:0/1708416593 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f499c011aa0 con 0x7f49ac0724e0 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.704+0000 7f49aaffd700 1 -- 172.21.15.78:0/1708416593 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f499c010cf0 con 0x7f49ac0724e0 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.704+0000 7f49aaffd700 1 -- 172.21.15.78:0/1708416593 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f499c01bda0 con 0x7f49ac0724e0 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.704+0000 7f49b4348700 1 -- 172.21.15.78:0/1708416593 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac10d400 con 0x7f49ac0724e0 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.704+0000 7f49b4348700 1 -- 172.21.15.78:0/1708416593 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac10d810 con 0x7f49ac0724e0 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.704+0000 7f49aaffd700 1 -- 172.21.15.78:0/1708416593 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f499c00f070 con 0x7f49ac0724e0 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.705+0000 7f49aaffd700 1 --2- 172.21.15.78:0/1708416593 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f499802fcf0 0x7f49980321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.705+0000 7f49aaffd700 1 -- 172.21.15.78:0/1708416593 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f499c041a20 con 0x7f49ac0724e0 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.705+0000 7f49b4348700 1 -- 172.21.15.78:0/1708416593 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac101e80 con 0x7f49ac0724e0 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.705+0000 7f49b18e3700 1 --2- 172.21.15.78:0/1708416593 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f499802fcf0 0x7f49980321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.705+0000 7f49b18e3700 1 --2- 172.21.15.78:0/1708416593 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f499802fcf0 0x7f49980321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f49a00096a0 tx=0x7f49a0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:20.711 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.709+0000 7f49aaffd700 1 -- 172.21.15.78:0/1708416593 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f499c024400 con 0x7f49ac0724e0 2023-12-21T03:39:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.825+0000 7f49aaffd700 1 -- 172.21.15.78:0/1708416593 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f499c01b440 con 0x7f49ac0724e0 2023-12-21T03:39:20.895 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.893+0000 7f49b4348700 1 -- 172.21.15.78:0/1708416593 --> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f49ac102040 con 0x7f499802fcf0 2023-12-21T03:39:20.898 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.897+0000 7f49aaffd700 1 -- 172.21.15.78:0/1708416593 <== mgr.14142 v2:172.21.15.78:6800/1046721827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f49ac102040 con 0x7f499802fcf0 2023-12-21T03:39:20.898 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-12-21T03:39:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.898+0000 7f49b4348700 1 -- 172.21.15.78:0/1708416593 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f499802fcf0 msgr2=0x7f49980321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.898+0000 7f49b4348700 1 --2- 172.21.15.78:0/1708416593 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f499802fcf0 0x7f49980321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f49a00096a0 tx=0x7f49a0006b40).stop 2023-12-21T03:39:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.898+0000 7f49b4348700 1 -- 172.21.15.78:0/1708416593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 msgr2=0x7f49ac10cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.898+0000 7f49b4348700 1 --2- 172.21.15.78:0/1708416593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac10cd10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f499c011cf0 tx=0x7f499c0092d0).stop 2023-12-21T03:39:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.898+0000 7f49b4348700 1 -- 172.21.15.78:0/1708416593 shutdown_connections 2023-12-21T03:39:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.898+0000 7f49b4348700 1 --2- 172.21.15.78:0/1708416593 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f499802fcf0 0x7f49980321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.898+0000 7f49b4348700 1 --2- 172.21.15.78:0/1708416593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0724e0 0x7f49ac10cd10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.898+0000 7f49b4348700 1 -- 172.21.15.78:0/1708416593 >> 172.21.15.78:0/1708416593 conn(0x7f49ac0f74b0 msgr2=0x7f49ac0f8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.898+0000 7f49b4348700 1 -- 172.21.15.78:0/1708416593 shutdown_connections 2023-12-21T03:39:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:20.899+0000 7f49b4348700 1 -- 172.21.15.78:0/1708416593 wait complete. 2023-12-21T03:39:21.056 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:20 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:19.821250+0000 mon.smithi078 (mon.0) 79 : cluster [DBG] mgrmap e11: smithi078.zidbgq(active, since 1.05801s) 2023-12-21T03:39:21.056 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:20 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:19.822321+0000 mgr.smithi078.zidbgq (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-12-21T03:39:21.057 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:20 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:19.825465+0000 mon.smithi078 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:Deploying node-exporter service with default placement... 2023-12-21T03:39:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-12-21T03:39:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.582+0000 7f722a22e700 1 Processor -- start 2023-12-21T03:39:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.582+0000 7f722a22e700 1 -- start start 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.583+0000 7f722a22e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f72240fe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.583+0000 7f722a22e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72240fecf0 con 0x7f7224070430 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.583+0000 7f72237fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f72240fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.583+0000 7f72237fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f72240fe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38714/0 (socket says 172.21.15.78:38714) 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.583+0000 7f72237fe700 1 -- 172.21.15.78:0/1192119223 learned_addr learned my addr 172.21.15.78:0/1192119223 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.583+0000 7f72237fe700 1 -- 172.21.15.78:0/1192119223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72240fee30 con 0x7f7224070430 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.583+0000 7f72237fe700 1 --2- 172.21.15.78:0/1192119223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f72240fe800 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f721800b5f0 tx=0x7f7218006ee0).ready entity=mon.0 client_cookie=d9acb31f4b985560 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.583+0000 7f72227fc700 1 -- 172.21.15.78:0/1192119223 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f721800d600 con 0x7f7224070430 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.583+0000 7f72227fc700 1 -- 172.21.15.78:0/1192119223 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f721800dc10 con 0x7f7224070430 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.583+0000 7f72227fc700 1 -- 172.21.15.78:0/1192119223 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7218011cd0 con 0x7f7224070430 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.584+0000 7f722a22e700 1 -- 172.21.15.78:0/1192119223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 msgr2=0x7f72240fe800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.584+0000 7f722a22e700 1 --2- 172.21.15.78:0/1192119223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f72240fe800 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f721800b5f0 tx=0x7f7218006ee0).stop 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.584+0000 7f722a22e700 1 -- 172.21.15.78:0/1192119223 shutdown_connections 2023-12-21T03:39:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.584+0000 7f722a22e700 1 --2- 172.21.15.78:0/1192119223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f72240fe800 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.584+0000 7f722a22e700 1 -- 172.21.15.78:0/1192119223 >> 172.21.15.78:0/1192119223 conn(0x7f72240f62a0 msgr2=0x7f72240f86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.584+0000 7f722a22e700 1 -- 172.21.15.78:0/1192119223 shutdown_connections 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.584+0000 7f722a22e700 1 -- 172.21.15.78:0/1192119223 wait complete. 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.584+0000 7f722a22e700 1 Processor -- start 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.585+0000 7f722a22e700 1 -- start start 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.585+0000 7f722a22e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f722410a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.585+0000 7f722a22e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722410aed0 con 0x7f7224070430 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.585+0000 7f72237fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f722410a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.585+0000 7f72237fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f722410a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38718/0 (socket says 172.21.15.78:38718) 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.585+0000 7f72237fe700 1 -- 172.21.15.78:0/3483813681 learned_addr learned my addr 172.21.15.78:0/3483813681 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.585+0000 7f72237fe700 1 -- 172.21.15.78:0/3483813681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721800b200 con 0x7f7224070430 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.585+0000 7f72237fe700 1 --2- 172.21.15.78:0/3483813681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f722410a9e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7218006800 tx=0x7f7218006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.585+0000 7f720bfff700 1 -- 172.21.15.78:0/3483813681 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7218011df0 con 0x7f7224070430 2023-12-21T03:39:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.585+0000 7f720bfff700 1 -- 172.21.15.78:0/3483813681 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f721801b8c0 con 0x7f7224070430 2023-12-21T03:39:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.585+0000 7f722a22e700 1 -- 172.21.15.78:0/3483813681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722410b0d0 con 0x7f7224070430 2023-12-21T03:39:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.586+0000 7f720bfff700 1 -- 172.21.15.78:0/3483813681 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f721801a800 con 0x7f7224070430 2023-12-21T03:39:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.586+0000 7f722a22e700 1 -- 172.21.15.78:0/3483813681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722410b560 con 0x7f7224070430 2023-12-21T03:39:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.586+0000 7f720bfff700 1 -- 172.21.15.78:0/3483813681 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f721800f070 con 0x7f7224070430 2023-12-21T03:39:21.591 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.586+0000 7f720bfff700 1 --2- 172.21.15.78:0/3483813681 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f721002feb0 0x7f7210032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:21.591 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.586+0000 7f720bfff700 1 -- 172.21.15.78:0/3483813681 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7218040f80 con 0x7f7224070430 2023-12-21T03:39:21.591 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.587+0000 7f7222ffd700 1 --2- 172.21.15.78:0/3483813681 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f721002feb0 0x7f7210032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:21.591 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.587+0000 7f722a22e700 1 -- 172.21.15.78:0/3483813681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7224045ad0 con 0x7f7224070430 2023-12-21T03:39:21.591 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.587+0000 7f7222ffd700 1 --2- 172.21.15.78:0/3483813681 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f721002feb0 0x7f7210032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f72140096a0 tx=0x7f7214006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:21.592 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.591+0000 7f720bfff700 1 -- 172.21.15.78:0/3483813681 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7218016070 con 0x7f7224070430 2023-12-21T03:39:21.769 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.767+0000 7f722a22e700 1 -- 172.21.15.78:0/3483813681 --> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f72241049c0 con 0x7f721002feb0 2023-12-21T03:39:21.772 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.771+0000 7f720bfff700 1 -- 172.21.15.78:0/3483813681 <== mgr.14142 v2:172.21.15.78:6800/1046721827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f72241049c0 con 0x7f721002feb0 2023-12-21T03:39:21.773 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-12-21T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.773+0000 7f722a22e700 1 -- 172.21.15.78:0/3483813681 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f721002feb0 msgr2=0x7f7210032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.773+0000 7f722a22e700 1 --2- 172.21.15.78:0/3483813681 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f721002feb0 0x7f7210032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f72140096a0 tx=0x7f7214006b40).stop 2023-12-21T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.773+0000 7f722a22e700 1 -- 172.21.15.78:0/3483813681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 msgr2=0x7f722410a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.773+0000 7f722a22e700 1 --2- 172.21.15.78:0/3483813681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f722410a9e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7218006800 tx=0x7f7218006ee0).stop 2023-12-21T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.773+0000 7f722a22e700 1 -- 172.21.15.78:0/3483813681 shutdown_connections 2023-12-21T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.773+0000 7f722a22e700 1 --2- 172.21.15.78:0/3483813681 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f721002feb0 0x7f7210032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.773+0000 7f722a22e700 1 --2- 172.21.15.78:0/3483813681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7224070430 0x7f722410a9e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.773+0000 7f722a22e700 1 -- 172.21.15.78:0/3483813681 >> 172.21.15.78:0/3483813681 conn(0x7f72240f62a0 msgr2=0x7f72240f7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.773+0000 7f722a22e700 1 -- 172.21.15.78:0/3483813681 shutdown_connections 2023-12-21T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:21.773+0000 7f722a22e700 1 -- 172.21.15.78:0/3483813681 wait complete. 2023-12-21T03:39:22.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:20.827554+0000 mon.smithi078 (mon.0) 81 : cluster [DBG] mgrmap e12: smithi078.zidbgq(active, since 2s) 2023-12-21T03:39:22.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:20.895321+0000 mgr.smithi078.zidbgq (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:22.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:20.895809+0000 mgr.smithi078.zidbgq (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-12-21T03:39:22.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:20.897913+0000 mon.smithi078 (mon.0) 82 : audit [INF] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:22.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:21.772007+0000 mon.smithi078 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:22.125 INFO:teuthology.orchestra.run.smithi078.stderr:Deploying alertmanager service with default placement... 2023-12-21T03:39:22.125 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-12-21T03:39:22.478 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.473+0000 7f9042e96700 1 Processor -- start 2023-12-21T03:39:22.478 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.473+0000 7f9042e96700 1 -- start start 2023-12-21T03:39:22.478 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.473+0000 7f9042e96700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c100d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.473+0000 7f9042e96700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0fc760 con 0x7f903c070420 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.473+0000 7f9040c32700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c100d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.473+0000 7f9040c32700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c100d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38730/0 (socket says 172.21.15.78:38730) 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.473+0000 7f9040c32700 1 -- 172.21.15.78:0/2387504732 learned_addr learned my addr 172.21.15.78:0/2387504732 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.473+0000 7f9040c32700 1 -- 172.21.15.78:0/2387504732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c101260 con 0x7f903c070420 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.473+0000 7f9040c32700 1 --2- 172.21.15.78:0/2387504732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c100d70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f903000b5f0 tx=0x7f9030006ee0).ready entity=mon.0 client_cookie=2492b1ce2364785c server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.474+0000 7f903b7fe700 1 -- 172.21.15.78:0/2387504732 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f903000d600 con 0x7f903c070420 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.474+0000 7f903b7fe700 1 -- 172.21.15.78:0/2387504732 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f903000dc10 con 0x7f903c070420 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.474+0000 7f903b7fe700 1 -- 172.21.15.78:0/2387504732 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9030011cd0 con 0x7f903c070420 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.474+0000 7f9042e96700 1 -- 172.21.15.78:0/2387504732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 msgr2=0x7f903c100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.474+0000 7f9042e96700 1 --2- 172.21.15.78:0/2387504732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c100d70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f903000b5f0 tx=0x7f9030006ee0).stop 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.474+0000 7f9042e96700 1 -- 172.21.15.78:0/2387504732 shutdown_connections 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.474+0000 7f9042e96700 1 --2- 172.21.15.78:0/2387504732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c100d70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:22.479 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.474+0000 7f9042e96700 1 -- 172.21.15.78:0/2387504732 >> 172.21.15.78:0/2387504732 conn(0x7f903c0f63b0 msgr2=0x7f903c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.475+0000 7f9042e96700 1 -- 172.21.15.78:0/2387504732 shutdown_connections 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.475+0000 7f9042e96700 1 -- 172.21.15.78:0/2387504732 wait complete. 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.475+0000 7f9042e96700 1 Processor -- start 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.475+0000 7f9042e96700 1 -- start start 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.475+0000 7f9042e96700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c10ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.475+0000 7f9042e96700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c10d1b0 con 0x7f903c070420 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.475+0000 7f9040c32700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c10ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.475+0000 7f9040c32700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c10ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38746/0 (socket says 172.21.15.78:38746) 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.475+0000 7f9040c32700 1 -- 172.21.15.78:0/104043608 learned_addr learned my addr 172.21.15.78:0/104043608 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.476+0000 7f9040c32700 1 -- 172.21.15.78:0/104043608 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903000b200 con 0x7f903c070420 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.476+0000 7f9040c32700 1 --2- 172.21.15.78:0/104043608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c10ccc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f903000b1d0 tx=0x7f9030006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.476+0000 7f90397fa700 1 -- 172.21.15.78:0/104043608 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9030011df0 con 0x7f903c070420 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.476+0000 7f9042e96700 1 -- 172.21.15.78:0/104043608 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c10d3b0 con 0x7f903c070420 2023-12-21T03:39:22.480 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.476+0000 7f90397fa700 1 -- 172.21.15.78:0/104043608 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f903001b8c0 con 0x7f903c070420 2023-12-21T03:39:22.481 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.476+0000 7f90397fa700 1 -- 172.21.15.78:0/104043608 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f903001a830 con 0x7f903c070420 2023-12-21T03:39:22.481 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.476+0000 7f9042e96700 1 -- 172.21.15.78:0/104043608 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c10d7c0 con 0x7f903c070420 2023-12-21T03:39:22.481 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.477+0000 7f90397fa700 1 -- 172.21.15.78:0/104043608 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f903000f070 con 0x7f903c070420 2023-12-21T03:39:22.482 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.477+0000 7f9042e96700 1 -- 172.21.15.78:0/104043608 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903c045ad0 con 0x7f903c070420 2023-12-21T03:39:22.482 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.477+0000 7f90397fa700 1 --2- 172.21.15.78:0/104043608 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f902002fe60 0x7f9020032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:22.482 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.477+0000 7f90397fa700 1 -- 172.21.15.78:0/104043608 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9030040fc0 con 0x7f903c070420 2023-12-21T03:39:22.482 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.477+0000 7f903bfff700 1 --2- 172.21.15.78:0/104043608 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f902002fe60 0x7f9020032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:22.482 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.478+0000 7f903bfff700 1 --2- 172.21.15.78:0/104043608 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f902002fe60 0x7f9020032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f90280096a0 tx=0x7f9028006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:22.483 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.481+0000 7f90397fa700 1 -- 172.21.15.78:0/104043608 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f903006d2b0 con 0x7f903c070420 2023-12-21T03:39:22.658 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.656+0000 7f9042e96700 1 -- 172.21.15.78:0/104043608 --> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f903c10e020 con 0x7f902002fe60 2023-12-21T03:39:22.661 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.660+0000 7f90397fa700 1 -- 172.21.15.78:0/104043608 <== mgr.14142 v2:172.21.15.78:6800/1046721827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f903c10e020 con 0x7f902002fe60 2023-12-21T03:39:22.661 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-12-21T03:39:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.661+0000 7f9042e96700 1 -- 172.21.15.78:0/104043608 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f902002fe60 msgr2=0x7f9020032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.661+0000 7f9042e96700 1 --2- 172.21.15.78:0/104043608 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f902002fe60 0x7f9020032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f90280096a0 tx=0x7f9028006b40).stop 2023-12-21T03:39:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.661+0000 7f9042e96700 1 -- 172.21.15.78:0/104043608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 msgr2=0x7f903c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.661+0000 7f9042e96700 1 --2- 172.21.15.78:0/104043608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c10ccc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f903000b1d0 tx=0x7f9030006ee0).stop 2023-12-21T03:39:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.661+0000 7f9042e96700 1 -- 172.21.15.78:0/104043608 shutdown_connections 2023-12-21T03:39:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.661+0000 7f9042e96700 1 --2- 172.21.15.78:0/104043608 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f902002fe60 0x7f9020032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.661+0000 7f9042e96700 1 --2- 172.21.15.78:0/104043608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f903c070420 0x7f903c10ccc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.661+0000 7f9042e96700 1 -- 172.21.15.78:0/104043608 >> 172.21.15.78:0/104043608 conn(0x7f903c0f63b0 msgr2=0x7f903c0f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.662+0000 7f9042e96700 1 -- 172.21.15.78:0/104043608 shutdown_connections 2023-12-21T03:39:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:22.662+0000 7f9042e96700 1 -- 172.21.15.78:0/104043608 wait complete. 2023-12-21T03:39:23.001 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:21.769233+0000 mgr.smithi078.zidbgq (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:23.001 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:21.769885+0000 mgr.smithi078.zidbgq (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-12-21T03:39:23.001 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:22.660823+0000 mon.smithi078 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.78:0/4150705244' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:23.041 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-12-21T03:39:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.379+0000 7f58070d1700 1 Processor -- start 2023-12-21T03:39:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.379+0000 7f58070d1700 1 -- start start 2023-12-21T03:39:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.379+0000 7f58070d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f58000fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.379+0000 7f58070d1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58000fb490 con 0x7f58000fab80 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.379+0000 7f5804e6d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f58000fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.379+0000 7f5804e6d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f58000fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38758/0 (socket says 172.21.15.78:38758) 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.379+0000 7f5804e6d700 1 -- 172.21.15.78:0/107916009 learned_addr learned my addr 172.21.15.78:0/107916009 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.380+0000 7f5804e6d700 1 -- 172.21.15.78:0/107916009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58000fb5d0 con 0x7f58000fab80 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.380+0000 7f5804e6d700 1 --2- 172.21.15.78:0/107916009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f58000fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f57f400b870 tx=0x7f57f4006f70).ready entity=mon.0 client_cookie=fad2cf58610e0c80 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.380+0000 7f57ff7fe700 1 -- 172.21.15.78:0/107916009 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57f400da20 con 0x7f58000fab80 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.380+0000 7f57ff7fe700 1 -- 172.21.15.78:0/107916009 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57f4010470 con 0x7f58000fab80 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.380+0000 7f57ff7fe700 1 -- 172.21.15.78:0/107916009 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57f400f5b0 con 0x7f58000fab80 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.380+0000 7f58070d1700 1 -- 172.21.15.78:0/107916009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 msgr2=0x7f58000fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.380+0000 7f58070d1700 1 --2- 172.21.15.78:0/107916009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f58000fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f57f400b870 tx=0x7f57f4006f70).stop 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.380+0000 7f58070d1700 1 -- 172.21.15.78:0/107916009 shutdown_connections 2023-12-21T03:39:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.380+0000 7f58070d1700 1 --2- 172.21.15.78:0/107916009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f58000fafa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.380+0000 7f58070d1700 1 -- 172.21.15.78:0/107916009 >> 172.21.15.78:0/107916009 conn(0x7f58000f63c0 msgr2=0x7f58000f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.380+0000 7f58070d1700 1 -- 172.21.15.78:0/107916009 shutdown_connections 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.381+0000 7f58070d1700 1 -- 172.21.15.78:0/107916009 wait complete. 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.381+0000 7f58070d1700 1 Processor -- start 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.381+0000 7f58070d1700 1 -- start start 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f58070d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f5800108890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f58070d1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5800108d80 con 0x7f58000fab80 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f5804e6d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f5800108890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f5804e6d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f5800108890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38774/0 (socket says 172.21.15.78:38774) 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f5804e6d700 1 -- 172.21.15.78:0/4235740765 learned_addr learned my addr 172.21.15.78:0/4235740765 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f5804e6d700 1 -- 172.21.15.78:0/4235740765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f400b480 con 0x7f58000fab80 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f5804e6d700 1 --2- 172.21.15.78:0/4235740765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f5800108890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f57f400b840 tx=0x7f57f4006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f57fd7fa700 1 -- 172.21.15.78:0/4235740765 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57f400f3d0 con 0x7f58000fab80 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f57fd7fa700 1 -- 172.21.15.78:0/4235740765 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57f400fef0 con 0x7f58000fab80 2023-12-21T03:39:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f57fd7fa700 1 -- 172.21.15.78:0/4235740765 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57f4019c70 con 0x7f58000fab80 2023-12-21T03:39:23.387 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f58070d1700 1 -- 172.21.15.78:0/4235740765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5800108f80 con 0x7f58000fab80 2023-12-21T03:39:23.387 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.382+0000 7f58070d1700 1 -- 172.21.15.78:0/4235740765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5800109390 con 0x7f58000fab80 2023-12-21T03:39:23.387 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.383+0000 7f57fd7fa700 1 -- 172.21.15.78:0/4235740765 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f57f4016030 con 0x7f58000fab80 2023-12-21T03:39:23.388 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.383+0000 7f58070d1700 1 -- 172.21.15.78:0/4235740765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5800045ad0 con 0x7f58000fab80 2023-12-21T03:39:23.388 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.383+0000 7f57fd7fa700 1 --2- 172.21.15.78:0/4235740765 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f57e402fe60 0x7f57e4032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:23.388 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.383+0000 7f57fd7fa700 1 -- 172.21.15.78:0/4235740765 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f57f40446a0 con 0x7f58000fab80 2023-12-21T03:39:23.388 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.384+0000 7f57fffff700 1 --2- 172.21.15.78:0/4235740765 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f57e402fe60 0x7f57e4032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:23.388 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.384+0000 7f57fffff700 1 --2- 172.21.15.78:0/4235740765 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f57e402fe60 0x7f57e4032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f57ec0096a0 tx=0x7f57ec006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:23.389 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.387+0000 7f57fd7fa700 1 -- 172.21.15.78:0/4235740765 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57f4022370 con 0x7f58000fab80 2023-12-21T03:39:23.538 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.536+0000 7f58070d1700 1 -- 172.21.15.78:0/4235740765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f58000fc900 con 0x7f58000fab80 2023-12-21T03:39:23.540 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.538+0000 7f57fd7fa700 1 -- 172.21.15.78:0/4235740765 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f57f402cb80 con 0x7f58000fab80 2023-12-21T03:39:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.540+0000 7f58070d1700 1 -- 172.21.15.78:0/4235740765 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f57e402fe60 msgr2=0x7f57e4032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.540+0000 7f58070d1700 1 --2- 172.21.15.78:0/4235740765 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f57e402fe60 0x7f57e4032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f57ec0096a0 tx=0x7f57ec006b40).stop 2023-12-21T03:39:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.540+0000 7f58070d1700 1 -- 172.21.15.78:0/4235740765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 msgr2=0x7f5800108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.540+0000 7f58070d1700 1 --2- 172.21.15.78:0/4235740765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f5800108890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f57f400b840 tx=0x7f57f4006f70).stop 2023-12-21T03:39:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.540+0000 7f58070d1700 1 -- 172.21.15.78:0/4235740765 shutdown_connections 2023-12-21T03:39:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.540+0000 7f58070d1700 1 --2- 172.21.15.78:0/4235740765 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f57e402fe60 0x7f57e4032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.540+0000 7f58070d1700 1 --2- 172.21.15.78:0/4235740765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58000fab80 0x7f5800108890 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.540+0000 7f58070d1700 1 -- 172.21.15.78:0/4235740765 >> 172.21.15.78:0/4235740765 conn(0x7f58000f63c0 msgr2=0x7f58000ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:23.544 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.541+0000 7f58070d1700 1 -- 172.21.15.78:0/4235740765 shutdown_connections 2023-12-21T03:39:23.544 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:23.541+0000 7f58070d1700 1 -- 172.21.15.78:0/4235740765 wait complete. 2023-12-21T03:39:23.890 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-12-21T03:39:24.252 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.249+0000 7f52c8348700 1 Processor -- start 2023-12-21T03:39:24.252 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.249+0000 7f52c8348700 1 -- start start 2023-12-21T03:39:24.252 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.249+0000 7f52c8348700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:24.252 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.249+0000 7f52c8348700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c00fc5c0 con 0x7f52c00fbcb0 2023-12-21T03:39:24.253 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.249+0000 7f52c60e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:24.253 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.249+0000 7f52c60e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38790/0 (socket says 172.21.15.78:38790) 2023-12-21T03:39:24.253 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.249+0000 7f52c60e4700 1 -- 172.21.15.78:0/1109329478 learned_addr learned my addr 172.21.15.78:0/1109329478 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:24.253 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.250+0000 7f52c60e4700 1 -- 172.21.15.78:0/1109329478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c00fc700 con 0x7f52c00fbcb0 2023-12-21T03:39:24.253 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.250+0000 7f52c60e4700 1 --2- 172.21.15.78:0/1109329478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00fc0d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f52b000b5f0 tx=0x7f52b0006ee0).ready entity=mon.0 client_cookie=5f7d7de151b2303b server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:24.253 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.250+0000 7f52c50e2700 1 -- 172.21.15.78:0/1109329478 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52b000d600 con 0x7f52c00fbcb0 2023-12-21T03:39:24.253 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.250+0000 7f52c50e2700 1 -- 172.21.15.78:0/1109329478 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52b000dc10 con 0x7f52c00fbcb0 2023-12-21T03:39:24.253 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.250+0000 7f52c50e2700 1 -- 172.21.15.78:0/1109329478 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52b0011cd0 con 0x7f52c00fbcb0 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.251+0000 7f52c8348700 1 -- 172.21.15.78:0/1109329478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 msgr2=0x7f52c00fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.251+0000 7f52c8348700 1 --2- 172.21.15.78:0/1109329478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00fc0d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f52b000b5f0 tx=0x7f52b0006ee0).stop 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.251+0000 7f52c8348700 1 -- 172.21.15.78:0/1109329478 shutdown_connections 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.251+0000 7f52c8348700 1 --2- 172.21.15.78:0/1109329478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00fc0d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.251+0000 7f52c8348700 1 -- 172.21.15.78:0/1109329478 >> 172.21.15.78:0/1109329478 conn(0x7f52c00f74b0 msgr2=0x7f52c00f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.251+0000 7f52c8348700 1 -- 172.21.15.78:0/1109329478 shutdown_connections 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.251+0000 7f52c8348700 1 -- 172.21.15.78:0/1109329478 wait complete. 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.251+0000 7f52c8348700 1 Processor -- start 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.251+0000 7f52c8348700 1 -- start start 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.251+0000 7f52c8348700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.251+0000 7f52c8348700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c0070a60 con 0x7f52c00fbcb0 2023-12-21T03:39:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.252+0000 7f52c60e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.252+0000 7f52c60e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38796/0 (socket says 172.21.15.78:38796) 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.252+0000 7f52c60e4700 1 -- 172.21.15.78:0/715067995 learned_addr learned my addr 172.21.15.78:0/715067995 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.252+0000 7f52c60e4700 1 -- 172.21.15.78:0/715067995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b000b200 con 0x7f52c00fbcb0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.252+0000 7f52c60e4700 1 --2- 172.21.15.78:0/715067995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00725e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f52b000e040 tx=0x7f52b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.252+0000 7f52aeffd700 1 -- 172.21.15.78:0/715067995 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52b001b410 con 0x7f52c00fbcb0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.252+0000 7f52aeffd700 1 -- 172.21.15.78:0/715067995 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52b001ba20 con 0x7f52c00fbcb0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.252+0000 7f52c8348700 1 -- 172.21.15.78:0/715067995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c0070c00 con 0x7f52c00fbcb0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.252+0000 7f52aeffd700 1 -- 172.21.15.78:0/715067995 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52b001a910 con 0x7f52c00fbcb0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.252+0000 7f52c8348700 1 -- 172.21.15.78:0/715067995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c0071010 con 0x7f52c00fbcb0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.253+0000 7f52aeffd700 1 -- 172.21.15.78:0/715067995 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f52b000f070 con 0x7f52c00fbcb0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.253+0000 7f52aeffd700 1 --2- 172.21.15.78:0/715067995 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f52a802feb0 0x7f52a8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.253+0000 7f52aeffd700 1 -- 172.21.15.78:0/715067995 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f52b0041160 con 0x7f52c00fbcb0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.253+0000 7f52c8348700 1 -- 172.21.15.78:0/715067995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c0045ad0 con 0x7f52c00fbcb0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.253+0000 7f52c58e3700 1 --2- 172.21.15.78:0/715067995 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f52a802feb0 0x7f52a8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.254+0000 7f52c58e3700 1 --2- 172.21.15.78:0/715067995 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f52a802feb0 0x7f52a8032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f52bc0096a0 tx=0x7f52bc006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:24.259 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.258+0000 7f52aeffd700 1 -- 172.21.15.78:0/715067995 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52b0024410 con 0x7f52c00fbcb0 2023-12-21T03:39:24.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:22.658154+0000 mgr.smithi078.zidbgq (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:24.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:22.658701+0000 mgr.smithi078.zidbgq (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-12-21T03:39:24.317 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:23.539950+0000 mon.smithi078 (mon.0) 85 : audit [INF] from='client.? 172.21.15.78:0/4235740765' entity='client.admin' 2023-12-21T03:39:24.407 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.405+0000 7f52c8348700 1 -- 172.21.15.78:0/715067995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f52c0027a00 con 0x7f52c00fbcb0 2023-12-21T03:39:24.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.408+0000 7f52aeffd700 1 -- 172.21.15.78:0/715067995 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f52b0006a70 con 0x7f52c00fbcb0 2023-12-21T03:39:24.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.410+0000 7f52c8348700 1 -- 172.21.15.78:0/715067995 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f52a802feb0 msgr2=0x7f52a8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:24.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.410+0000 7f52c8348700 1 --2- 172.21.15.78:0/715067995 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f52a802feb0 0x7f52a8032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f52bc0096a0 tx=0x7f52bc006b40).stop 2023-12-21T03:39:24.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.410+0000 7f52c8348700 1 -- 172.21.15.78:0/715067995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 msgr2=0x7f52c00725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:24.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.410+0000 7f52c8348700 1 --2- 172.21.15.78:0/715067995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00725e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f52b000e040 tx=0x7f52b0006ee0).stop 2023-12-21T03:39:24.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.410+0000 7f52c8348700 1 -- 172.21.15.78:0/715067995 shutdown_connections 2023-12-21T03:39:24.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.410+0000 7f52c8348700 1 --2- 172.21.15.78:0/715067995 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f52a802feb0 0x7f52a8032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:24.414 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.410+0000 7f52c8348700 1 --2- 172.21.15.78:0/715067995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52c00fbcb0 0x7f52c00725e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:24.414 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.410+0000 7f52c8348700 1 -- 172.21.15.78:0/715067995 >> 172.21.15.78:0/715067995 conn(0x7f52c00f74b0 msgr2=0x7f52c00f8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:24.414 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.410+0000 7f52c8348700 1 -- 172.21.15.78:0/715067995 shutdown_connections 2023-12-21T03:39:24.415 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:24.411+0000 7f52c8348700 1 -- 172.21.15.78:0/715067995 wait complete. 2023-12-21T03:39:24.774 INFO:teuthology.orchestra.run.smithi078.stderr:Enabling the dashboard module... 2023-12-21T03:39:24.774 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-12-21T03:39:25.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.127+0000 7f400012d700 1 Processor -- start 2023-12-21T03:39:25.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.127+0000 7f400012d700 1 -- start start 2023-12-21T03:39:25.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.127+0000 7f400012d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff80fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:25.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.127+0000 7f400012d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff80fedd0 con 0x7f3ff8070430 2023-12-21T03:39:25.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.127+0000 7f3ffdec9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff80fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:25.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.127+0000 7f3ffdec9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff80fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38804/0 (socket says 172.21.15.78:38804) 2023-12-21T03:39:25.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.128+0000 7f3ffdec9700 1 -- 172.21.15.78:0/486063538 learned_addr learned my addr 172.21.15.78:0/486063538 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:25.131 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.128+0000 7f3ffdec9700 1 -- 172.21.15.78:0/486063538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff80fef10 con 0x7f3ff8070430 2023-12-21T03:39:25.131 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.128+0000 7f3ffdec9700 1 --2- 172.21.15.78:0/486063538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff80fe8e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f3fe800b5f0 tx=0x7f3fe8006ee0).ready entity=mon.0 client_cookie=d7d501d963884340 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:25.131 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.128+0000 7f3ffcec7700 1 -- 172.21.15.78:0/486063538 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe800d600 con 0x7f3ff8070430 2023-12-21T03:39:25.131 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.128+0000 7f3ffcec7700 1 -- 172.21.15.78:0/486063538 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fe800dc10 con 0x7f3ff8070430 2023-12-21T03:39:25.131 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.128+0000 7f3ffcec7700 1 -- 172.21.15.78:0/486063538 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe8011cd0 con 0x7f3ff8070430 2023-12-21T03:39:25.133 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.129+0000 7f400012d700 1 -- 172.21.15.78:0/486063538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 msgr2=0x7f3ff80fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.129+0000 7f400012d700 1 --2- 172.21.15.78:0/486063538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff80fe8e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f3fe800b5f0 tx=0x7f3fe8006ee0).stop 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.129+0000 7f400012d700 1 -- 172.21.15.78:0/486063538 shutdown_connections 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.129+0000 7f400012d700 1 --2- 172.21.15.78:0/486063538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff80fe8e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.129+0000 7f400012d700 1 -- 172.21.15.78:0/486063538 >> 172.21.15.78:0/486063538 conn(0x7f3ff80f63c0 msgr2=0x7f3ff80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.129+0000 7f400012d700 1 -- 172.21.15.78:0/486063538 shutdown_connections 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.129+0000 7f400012d700 1 -- 172.21.15.78:0/486063538 wait complete. 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.129+0000 7f400012d700 1 Processor -- start 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f400012d700 1 -- start start 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f400012d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff810aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f400012d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff810afd0 con 0x7f3ff8070430 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f3ffdec9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff810aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f3ffdec9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff810aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38812/0 (socket says 172.21.15.78:38812) 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f3ffdec9700 1 -- 172.21.15.78:0/323070686 learned_addr learned my addr 172.21.15.78:0/323070686 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:25.134 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f3ffdec9700 1 -- 172.21.15.78:0/323070686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe800b200 con 0x7f3ff8070430 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f3ffdec9700 1 --2- 172.21.15.78:0/323070686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff810aae0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f3fe8006800 tx=0x7f3fe8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f3fe67fc700 1 -- 172.21.15.78:0/323070686 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe8011df0 con 0x7f3ff8070430 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f3fe67fc700 1 -- 172.21.15.78:0/323070686 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fe801b8c0 con 0x7f3ff8070430 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f3fe67fc700 1 -- 172.21.15.78:0/323070686 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe801a830 con 0x7f3ff8070430 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.130+0000 7f400012d700 1 -- 172.21.15.78:0/323070686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff810b1d0 con 0x7f3ff8070430 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.131+0000 7f400012d700 1 -- 172.21.15.78:0/323070686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff810b5e0 con 0x7f3ff8070430 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.131+0000 7f3fe67fc700 1 -- 172.21.15.78:0/323070686 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f3fe800f070 con 0x7f3ff8070430 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.131+0000 7f3fe67fc700 1 --2- 172.21.15.78:0/323070686 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f3fe002feb0 0x7f3fe0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.131+0000 7f3fe67fc700 1 -- 172.21.15.78:0/323070686 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f3fe8040ff0 con 0x7f3ff8070430 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.131+0000 7f3ffd6c8700 1 --2- 172.21.15.78:0/323070686 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f3fe002feb0 0x7f3fe0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.132+0000 7f400012d700 1 -- 172.21.15.78:0/323070686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff8059100 con 0x7f3ff8070430 2023-12-21T03:39:25.135 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.132+0000 7f3ffd6c8700 1 --2- 172.21.15.78:0/323070686 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f3fe002feb0 0x7f3fe0032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3ff40096a0 tx=0x7f3ff4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:25.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.136+0000 7f3fe67fc700 1 -- 172.21.15.78:0/323070686 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fe8016080 con 0x7f3ff8070430 2023-12-21T03:39:25.323 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.321+0000 7f400012d700 1 -- 172.21.15.78:0/323070686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f3ff80ffc00 con 0x7f3ff8070430 2023-12-21T03:39:25.414 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.412+0000 7f3fe67fc700 1 -- 172.21.15.78:0/323070686 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3fe801a990 con 0x7f3ff8070430 2023-12-21T03:39:25.414 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.413+0000 7f3fe67fc700 1 -- 172.21.15.78:0/323070686 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f3fe802c7f0 con 0x7f3ff8070430 2023-12-21T03:39:25.418 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.414+0000 7f400012d700 1 -- 172.21.15.78:0/323070686 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f3fe002feb0 msgr2=0x7f3fe0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:25.418 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.414+0000 7f400012d700 1 --2- 172.21.15.78:0/323070686 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f3fe002feb0 0x7f3fe0032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3ff40096a0 tx=0x7f3ff4006b40).stop 2023-12-21T03:39:25.418 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.414+0000 7f400012d700 1 -- 172.21.15.78:0/323070686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 msgr2=0x7f3ff810aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:25.418 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.414+0000 7f400012d700 1 --2- 172.21.15.78:0/323070686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff810aae0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f3fe8006800 tx=0x7f3fe8006ee0).stop 2023-12-21T03:39:25.418 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.414+0000 7f400012d700 1 -- 172.21.15.78:0/323070686 shutdown_connections 2023-12-21T03:39:25.418 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.414+0000 7f400012d700 1 --2- 172.21.15.78:0/323070686 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7f3fe002feb0 0x7f3fe0032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:25.419 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.414+0000 7f400012d700 1 --2- 172.21.15.78:0/323070686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff8070430 0x7f3ff810aae0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:25.419 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.415+0000 7f400012d700 1 -- 172.21.15.78:0/323070686 >> 172.21.15.78:0/323070686 conn(0x7f3ff80f63c0 msgr2=0x7f3ff80f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:25.419 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.415+0000 7f400012d700 1 -- 172.21.15.78:0/323070686 shutdown_connections 2023-12-21T03:39:25.419 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:25.415+0000 7f400012d700 1 -- 172.21.15.78:0/323070686 wait complete. 2023-12-21T03:39:25.677 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:24.409716+0000 mon.smithi078 (mon.0) 86 : audit [INF] from='client.? 172.21.15.78:0/715067995' entity='client.admin' 2023-12-21T03:39:25.677 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:25.323351+0000 mon.smithi078 (mon.0) 87 : audit [INF] from='client.? 172.21.15.78:0/323070686' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-21T03:39:25.774 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-21T03:39:26.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.135+0000 7fb39b59e700 1 Processor -- start 2023-12-21T03:39:26.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.135+0000 7fb39b59e700 1 -- start start 2023-12-21T03:39:26.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.135+0000 7fb39b59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:26.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.135+0000 7fb39b59e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c0fb2e0 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.135+0000 7fb39a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:26.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.135+0000 7fb39a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38844/0 (socket says 172.21.15.78:38844) 2023-12-21T03:39:26.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.135+0000 7fb39a59c700 1 -- 172.21.15.78:0/602825918 learned_addr learned my addr 172.21.15.78:0/602825918 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:26.138 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.136+0000 7fb39a59c700 1 -- 172.21.15.78:0/602825918 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c0fb420 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.139 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.136+0000 7fb39a59c700 1 --2- 172.21.15.78:0/602825918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c0fadf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fb38c00b5f0 tx=0x7fb38c006ee0).ready entity=mon.0 client_cookie=f262175bc2529f5b server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:26.139 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.136+0000 7fb39959a700 1 -- 172.21.15.78:0/602825918 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb38c00d600 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.139 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.136+0000 7fb39959a700 1 -- 172.21.15.78:0/602825918 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb38c00dc10 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.139 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.136+0000 7fb39959a700 1 -- 172.21.15.78:0/602825918 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb38c011cd0 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.140 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.137+0000 7fb39b59e700 1 -- 172.21.15.78:0/602825918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 msgr2=0x7fb39c0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.137+0000 7fb39b59e700 1 --2- 172.21.15.78:0/602825918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c0fadf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fb38c00b5f0 tx=0x7fb38c006ee0).stop 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.137+0000 7fb39b59e700 1 -- 172.21.15.78:0/602825918 shutdown_connections 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.137+0000 7fb39b59e700 1 --2- 172.21.15.78:0/602825918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c0fadf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.137+0000 7fb39b59e700 1 -- 172.21.15.78:0/602825918 >> 172.21.15.78:0/602825918 conn(0x7fb39c0f6210 msgr2=0x7fb39c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.137+0000 7fb39b59e700 1 -- 172.21.15.78:0/602825918 shutdown_connections 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.137+0000 7fb39b59e700 1 -- 172.21.15.78:0/602825918 wait complete. 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.138+0000 7fb39b59e700 1 Processor -- start 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.138+0000 7fb39b59e700 1 -- start start 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.138+0000 7fb39b59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c10d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.138+0000 7fb39b59e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c0704b0 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.138+0000 7fb39a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c10d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.138+0000 7fb39a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c10d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38850/0 (socket says 172.21.15.78:38850) 2023-12-21T03:39:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.138+0000 7fb39a59c700 1 -- 172.21.15.78:0/3192655431 learned_addr learned my addr 172.21.15.78:0/3192655431 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:26.142 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.138+0000 7fb39a59c700 1 -- 172.21.15.78:0/3192655431 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c00b200 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.142 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.138+0000 7fb39a59c700 1 --2- 172.21.15.78:0/3192655431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c10d5a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fb38c00dfb0 tx=0x7fb38c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:26.142 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.139+0000 7fb38affd700 1 -- 172.21.15.78:0/3192655431 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb38c011df0 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.142 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.139+0000 7fb39b59e700 1 -- 172.21.15.78:0/3192655431 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c0706b0 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.142 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.139+0000 7fb38affd700 1 -- 172.21.15.78:0/3192655431 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb38c01b8c0 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.142 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.139+0000 7fb38affd700 1 -- 172.21.15.78:0/3192655431 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb38c01a8c0 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.142 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.139+0000 7fb39b59e700 1 -- 172.21.15.78:0/3192655431 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c10dc50 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.143 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.139+0000 7fb38affd700 1 -- 172.21.15.78:0/3192655431 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fb38c00f070 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.143 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.140+0000 7fb38affd700 1 --2- 172.21.15.78:0/3192655431 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7fb38002feb0 0x7fb380032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:26.143 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.140+0000 7fb38affd700 1 -- 172.21.15.78:0/3192655431 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb38c0410f0 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.143 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.140+0000 7fb399d9b700 1 -- 172.21.15.78:0/3192655431 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7fb38002feb0 msgr2=0x7fb380032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1046721827 2023-12-21T03:39:26.143 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.140+0000 7fb399d9b700 1 --2- 172.21.15.78:0/3192655431 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7fb38002feb0 0x7fb380032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-21T03:39:26.143 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.140+0000 7fb39b59e700 1 -- 172.21.15.78:0/3192655431 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c045ad0 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.146 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.144+0000 7fb38affd700 1 -- 172.21.15.78:0/3192655431 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb38c016070 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.338 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.335+0000 7fb39b59e700 1 -- 172.21.15.78:0/3192655431 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb39c0fc7f0 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.338 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.336+0000 7fb38affd700 1 -- 172.21.15.78:0/3192655431 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fb38c010be0 con 0x7fb39c0fa9d0 2023-12-21T03:39:26.338 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-12-21T03:39:26.338 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 13, 2023-12-21T03:39:26.339 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "available": true, 2023-12-21T03:39:26.339 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "active_name": "smithi078.zidbgq", 2023-12-21T03:39:26.339 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-21T03:39:26.339 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-12-21T03:39:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.338+0000 7fb39b59e700 1 -- 172.21.15.78:0/3192655431 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7fb38002feb0 msgr2=0x7fb380032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-21T03:39:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.338+0000 7fb39b59e700 1 --2- 172.21.15.78:0/3192655431 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7fb38002feb0 0x7fb380032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:39:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.338+0000 7fb39b59e700 1 -- 172.21.15.78:0/3192655431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 msgr2=0x7fb39c10d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.338+0000 7fb39b59e700 1 --2- 172.21.15.78:0/3192655431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c10d5a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fb38c00dfb0 tx=0x7fb38c006ee0).stop 2023-12-21T03:39:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.338+0000 7fb39b59e700 1 -- 172.21.15.78:0/3192655431 shutdown_connections 2023-12-21T03:39:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.338+0000 7fb39b59e700 1 --2- 172.21.15.78:0/3192655431 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7fb38002feb0 0x7fb380032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:39:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.338+0000 7fb39b59e700 1 --2- 172.21.15.78:0/3192655431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c0fa9d0 0x7fb39c10d5a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.338+0000 7fb39b59e700 1 -- 172.21.15.78:0/3192655431 >> 172.21.15.78:0/3192655431 conn(0x7fb39c0f6210 msgr2=0x7fb39c0f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.338+0000 7fb39b59e700 1 -- 172.21.15.78:0/3192655431 shutdown_connections 2023-12-21T03:39:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:26.338+0000 7fb39b59e700 1 -- 172.21.15.78:0/3192655431 wait complete. 2023-12-21T03:39:26.626 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:25.414184+0000 mon.smithi078 (mon.0) 88 : audit [INF] from='client.? 172.21.15.78:0/323070686' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-21T03:39:26.626 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:25.414246+0000 mon.smithi078 (mon.0) 89 : cluster [DBG] mgrmap e13: smithi078.zidbgq(active, since 6s) 2023-12-21T03:39:26.626 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:26.337314+0000 mon.smithi078 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.78:0/3192655431' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-21T03:39:26.700 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for the mgr to restart... 2023-12-21T03:39:26.700 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mgr epoch 13... 2023-12-21T03:39:26.700 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-21T03:39:27.091 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.087+0000 7ffa099d5700 1 Processor -- start 2023-12-21T03:39:27.091 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.087+0000 7ffa099d5700 1 -- start start 2023-12-21T03:39:27.091 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.087+0000 7ffa099d5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa040fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:27.091 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.087+0000 7ffa099d5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa040fee30 con 0x7ffa040703d0 2023-12-21T03:39:27.091 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.087+0000 7ffa02ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa040fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.087+0000 7ffa02ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa040fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38860/0 (socket says 172.21.15.78:38860) 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.087+0000 7ffa02ffd700 1 -- 172.21.15.78:0/1975217505 learned_addr learned my addr 172.21.15.78:0/1975217505 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.087+0000 7ffa02ffd700 1 -- 172.21.15.78:0/1975217505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa040fef70 con 0x7ffa040703d0 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.087+0000 7ffa02ffd700 1 --2- 172.21.15.78:0/1975217505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa040fe940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff9f400b5f0 tx=0x7ff9f4006ee0).ready entity=mon.0 client_cookie=fa7629eadd9ef06 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.088+0000 7ffa01ffb700 1 -- 172.21.15.78:0/1975217505 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9f400d600 con 0x7ffa040703d0 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.088+0000 7ffa01ffb700 1 -- 172.21.15.78:0/1975217505 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9f400dc10 con 0x7ffa040703d0 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.088+0000 7ffa01ffb700 1 -- 172.21.15.78:0/1975217505 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9f4011cd0 con 0x7ffa040703d0 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.088+0000 7ffa099d5700 1 -- 172.21.15.78:0/1975217505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 msgr2=0x7ffa040fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.088+0000 7ffa099d5700 1 --2- 172.21.15.78:0/1975217505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa040fe940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff9f400b5f0 tx=0x7ff9f4006ee0).stop 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.088+0000 7ffa099d5700 1 -- 172.21.15.78:0/1975217505 shutdown_connections 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.088+0000 7ffa099d5700 1 --2- 172.21.15.78:0/1975217505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa040fe940 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.088+0000 7ffa099d5700 1 -- 172.21.15.78:0/1975217505 >> 172.21.15.78:0/1975217505 conn(0x7ffa040f63c0 msgr2=0x7ffa040f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.088+0000 7ffa099d5700 1 -- 172.21.15.78:0/1975217505 shutdown_connections 2023-12-21T03:39:27.092 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.088+0000 7ffa099d5700 1 -- 172.21.15.78:0/1975217505 wait complete. 2023-12-21T03:39:27.093 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.089+0000 7ffa099d5700 1 Processor -- start 2023-12-21T03:39:27.093 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.089+0000 7ffa099d5700 1 -- start start 2023-12-21T03:39:27.093 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.089+0000 7ffa099d5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa0410aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:27.093 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.089+0000 7ffa099d5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0410af30 con 0x7ffa040703d0 2023-12-21T03:39:27.096 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.089+0000 7ffa02ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa0410aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:27.096 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.090+0000 7ffa02ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa0410aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38876/0 (socket says 172.21.15.78:38876) 2023-12-21T03:39:27.096 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.090+0000 7ffa02ffd700 1 -- 172.21.15.78:0/4159063420 learned_addr learned my addr 172.21.15.78:0/4159063420 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:27.096 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.090+0000 7ffa02ffd700 1 -- 172.21.15.78:0/4159063420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f400b200 con 0x7ffa040703d0 2023-12-21T03:39:27.096 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.090+0000 7ffa02ffd700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa0410aa40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff9f4006a40 tx=0x7ff9f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.090+0000 7ff9eb7fe700 1 -- 172.21.15.78:0/4159063420 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9f4011c90 con 0x7ffa040703d0 2023-12-21T03:39:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.090+0000 7ff9eb7fe700 1 -- 172.21.15.78:0/4159063420 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9f401b8c0 con 0x7ffa040703d0 2023-12-21T03:39:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.090+0000 7ffa099d5700 1 -- 172.21.15.78:0/4159063420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa0410b130 con 0x7ffa040703d0 2023-12-21T03:39:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.090+0000 7ff9eb7fe700 1 -- 172.21.15.78:0/4159063420 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9f401a8c0 con 0x7ffa040703d0 2023-12-21T03:39:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.090+0000 7ffa099d5700 1 -- 172.21.15.78:0/4159063420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0410b540 con 0x7ffa040703d0 2023-12-21T03:39:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.091+0000 7ff9eb7fe700 1 -- 172.21.15.78:0/4159063420 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ff9f400f070 con 0x7ffa040703d0 2023-12-21T03:39:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.091+0000 7ff9eb7fe700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 0x7ff9f0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.091+0000 7ff9eb7fe700 1 -- 172.21.15.78:0/4159063420 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff9f4041a00 con 0x7ffa040703d0 2023-12-21T03:39:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.091+0000 7ffa099d5700 1 -- 172.21.15.78:0/4159063420 --> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa04059100 con 0x7ff9f002feb0 2023-12-21T03:39:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.091+0000 7ffa027fc700 1 -- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 msgr2=0x7ff9f0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1046721827 2023-12-21T03:39:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.091+0000 7ffa027fc700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 0x7ff9f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-21T03:39:27.294 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.292+0000 7ffa027fc700 1 -- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 msgr2=0x7ff9f0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1046721827 2023-12-21T03:39:27.294 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.292+0000 7ffa027fc700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 0x7ff9f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-21T03:39:27.695 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.693+0000 7ffa027fc700 1 -- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 msgr2=0x7ff9f0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1046721827 2023-12-21T03:39:27.698 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:27.693+0000 7ffa027fc700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 0x7ff9f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-21T03:39:28.496 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:28.494+0000 7ffa027fc700 1 -- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 msgr2=0x7ff9f0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1046721827 2023-12-21T03:39:28.496 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:28.494+0000 7ffa027fc700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 0x7ff9f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-21T03:39:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:30.095+0000 7ffa027fc700 1 -- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 msgr2=0x7ff9f0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1046721827 2023-12-21T03:39:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:30.095+0000 7ffa027fc700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 0x7ff9f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-21T03:39:30.400 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:30.398+0000 7ff9eb7fe700 1 -- 172.21.15.78:0/4159063420 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7ff9f40245a0 con 0x7ffa040703d0 2023-12-21T03:39:30.400 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:30.398+0000 7ff9eb7fe700 1 -- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 msgr2=0x7ff9f0032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-21T03:39:30.400 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:30.398+0000 7ff9eb7fe700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 0x7ff9f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:39:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.399+0000 7ff9eb7fe700 1 -- 172.21.15.78:0/4159063420 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7ff9f401b410 con 0x7ffa040703d0 2023-12-21T03:39:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.399+0000 7ff9eb7fe700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff9f0033990 0x7ff9f0035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.399+0000 7ff9eb7fe700 1 -- 172.21.15.78:0/4159063420 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa04059100 con 0x7ff9f0033990 2023-12-21T03:39:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.399+0000 7ffa027fc700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff9f0033990 0x7ff9f0035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.400+0000 7ffa027fc700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff9f0033990 0x7ff9f0035d80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7ff9f8006fd0 tx=0x7ff9f8006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.401+0000 7ff9eb7fe700 1 -- 172.21.15.78:0/4159063420 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7ffa04059100 con 0x7ff9f0033990 2023-12-21T03:39:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.406+0000 7ffa099d5700 1 -- 172.21.15.78:0/4159063420 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ffa04045ad0 con 0x7ff9f0033990 2023-12-21T03:39:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ff9eb7fe700 1 -- 172.21.15.78:0/4159063420 <== mgr.14162 v2:172.21.15.78:6800/2487427889 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7ffa04045ad0 con 0x7ff9f0033990 2023-12-21T03:39:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-12-21T03:39:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-12-21T03:39:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "initialized": true 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ffa099d5700 1 -- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff9f0033990 msgr2=0x7ff9f0035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ffa099d5700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff9f0033990 0x7ff9f0035d80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7ff9f8006fd0 tx=0x7ff9f8006cf0).stop 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ffa099d5700 1 -- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 msgr2=0x7ffa0410aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ffa099d5700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa0410aa40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff9f4006a40 tx=0x7ff9f4006ee0).stop 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ffa099d5700 1 -- 172.21.15.78:0/4159063420 shutdown_connections 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ffa099d5700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff9f0033990 0x7ff9f0035d80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ffa099d5700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa040703d0 0x7ffa0410aa40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ffa099d5700 1 --2- 172.21.15.78:0/4159063420 >> [v2:172.21.15.78:6800/1046721827,v1:172.21.15.78:6801/1046721827] conn(0x7ff9f002feb0 0x7ff9f0032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ffa099d5700 1 -- 172.21.15.78:0/4159063420 >> 172.21.15.78:0/4159063420 conn(0x7ffa040f63c0 msgr2=0x7ffa040f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ffa099d5700 1 -- 172.21.15.78:0/4159063420 shutdown_connections 2023-12-21T03:39:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:31.407+0000 7ffa099d5700 1 -- 172.21.15.78:0/4159063420 wait complete. 2023-12-21T03:39:31.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:30.343991+0000 mon.smithi078 (mon.0) 91 : cluster [INF] Active manager daemon smithi078.zidbgq restarted 2023-12-21T03:39:31.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:30.344524+0000 mon.smithi078 (mon.0) 92 : cluster [INF] Activating manager daemon smithi078.zidbgq 2023-12-21T03:39:31.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:30.348698+0000 mon.smithi078 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-21T03:39:31.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:30.399966+0000 mon.smithi078 (mon.0) 94 : cluster [DBG] mgrmap e14: smithi078.zidbgq(active, starting, since 0.055569s) 2023-12-21T03:39:31.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:30.401017+0000 mon.smithi078 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2023-12-21T03:39:31.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:30.401600+0000 mon.smithi078 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mgr metadata", "who": "smithi078.zidbgq", "id": "smithi078.zidbgq"}]: dispatch 2023-12-21T03:39:31.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:30.402232+0000 mon.smithi078 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-21T03:39:31.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:30.402447+0000 mon.smithi078 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-21T03:39:31.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:30.402653+0000 mon.smithi078 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-21T03:39:31.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:30.416775+0000 mon.smithi078 (mon.0) 100 : cluster [INF] Manager daemon smithi078.zidbgq is now available 2023-12-21T03:39:31.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:30.521052+0000 mon.smithi078 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:39:31.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:30.523264+0000 mon.smithi078 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:39:31.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:30.538730+0000 mon.smithi078 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:39:31.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:30.541153+0000 mon.smithi078 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:39:31.764 INFO:teuthology.orchestra.run.smithi078.stderr:mgr epoch 13 is available 2023-12-21T03:39:31.764 INFO:teuthology.orchestra.run.smithi078.stderr:Generating a dashboard self-signed certificate... 2023-12-21T03:39:31.764 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-12-21T03:39:32.128 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.124+0000 7f7a822e5700 1 Processor -- start 2023-12-21T03:39:32.128 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.124+0000 7f7a822e5700 1 -- start start 2023-12-21T03:39:32.128 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.124+0000 7f7a822e5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c070a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:32.128 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.124+0000 7f7a822e5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0728f0 con 0x7f7a7c0724d0 2023-12-21T03:39:32.128 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.125+0000 7f7a812e3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:32.128 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.125+0000 7f7a812e3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c070a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55124/0 (socket says 172.21.15.78:55124) 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.125+0000 7f7a812e3700 1 -- 172.21.15.78:0/3838737929 learned_addr learned my addr 172.21.15.78:0/3838737929 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.125+0000 7f7a812e3700 1 -- 172.21.15.78:0/3838737929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c070f60 con 0x7f7a7c0724d0 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.125+0000 7f7a812e3700 1 --2- 172.21.15.78:0/3838737929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c070a70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7a7800b5f0 tx=0x7f7a78006ee0).ready entity=mon.0 client_cookie=c137a6c4d47018fb server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.125+0000 7f7a6bfff700 1 -- 172.21.15.78:0/3838737929 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a7800d600 con 0x7f7a7c0724d0 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.125+0000 7f7a6bfff700 1 -- 172.21.15.78:0/3838737929 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a7800dc10 con 0x7f7a7c0724d0 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.125+0000 7f7a6bfff700 1 -- 172.21.15.78:0/3838737929 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a78011cd0 con 0x7f7a7c0724d0 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.126+0000 7f7a822e5700 1 -- 172.21.15.78:0/3838737929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 msgr2=0x7f7a7c070a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.126+0000 7f7a822e5700 1 --2- 172.21.15.78:0/3838737929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c070a70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7a7800b5f0 tx=0x7f7a78006ee0).stop 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.126+0000 7f7a822e5700 1 -- 172.21.15.78:0/3838737929 shutdown_connections 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.126+0000 7f7a822e5700 1 --2- 172.21.15.78:0/3838737929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c070a70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.126+0000 7f7a822e5700 1 -- 172.21.15.78:0/3838737929 >> 172.21.15.78:0/3838737929 conn(0x7f7a7c0f7380 msgr2=0x7f7a7c0f97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.126+0000 7f7a822e5700 1 -- 172.21.15.78:0/3838737929 shutdown_connections 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.126+0000 7f7a822e5700 1 -- 172.21.15.78:0/3838737929 wait complete. 2023-12-21T03:39:32.129 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a822e5700 1 Processor -- start 2023-12-21T03:39:32.130 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a822e5700 1 -- start start 2023-12-21T03:39:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a822e5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c10cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a822e5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c10d040 con 0x7f7a7c0724d0 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a812e3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c10cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a812e3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c10cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55136/0 (socket says 172.21.15.78:55136) 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a812e3700 1 -- 172.21.15.78:0/853675475 learned_addr learned my addr 172.21.15.78:0/853675475 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a812e3700 1 -- 172.21.15.78:0/853675475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7800b200 con 0x7f7a7c0724d0 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a812e3700 1 --2- 172.21.15.78:0/853675475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c10cb50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f7a78006800 tx=0x7f7a78006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a69ffb700 1 -- 172.21.15.78:0/853675475 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a78011df0 con 0x7f7a7c0724d0 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a69ffb700 1 -- 172.21.15.78:0/853675475 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a7801b8c0 con 0x7f7a7c0724d0 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a69ffb700 1 -- 172.21.15.78:0/853675475 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a7801a800 con 0x7f7a7c0724d0 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.127+0000 7f7a822e5700 1 -- 172.21.15.78:0/853675475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7c10d240 con 0x7f7a7c0724d0 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.128+0000 7f7a822e5700 1 -- 172.21.15.78:0/853675475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7c10d650 con 0x7f7a7c0724d0 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.129+0000 7f7a69ffb700 1 -- 172.21.15.78:0/853675475 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f7a7800f070 con 0x7f7a7c0724d0 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.129+0000 7f7a822e5700 1 -- 172.21.15.78:0/853675475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7c045ad0 con 0x7f7a7c0724d0 2023-12-21T03:39:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.129+0000 7f7a69ffb700 1 --2- 172.21.15.78:0/853675475 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a6402fd70 0x7f7a64032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:32.133 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.129+0000 7f7a69ffb700 1 -- 172.21.15.78:0/853675475 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a78040f60 con 0x7f7a7c0724d0 2023-12-21T03:39:32.133 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.130+0000 7f7a80ae2700 1 --2- 172.21.15.78:0/853675475 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a6402fd70 0x7f7a64032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:32.133 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.130+0000 7f7a80ae2700 1 --2- 172.21.15.78:0/853675475 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a6402fd70 0x7f7a64032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7a6c0096a0 tx=0x7f7a6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:32.136 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.134+0000 7f7a69ffb700 1 -- 172.21.15.78:0/853675475 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a7806d2b0 con 0x7f7a7c0724d0 2023-12-21T03:39:32.298 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.296+0000 7f7a822e5700 1 -- 172.21.15.78:0/853675475 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a7c10df90 con 0x7f7a6402fd70 2023-12-21T03:39:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.356+0000 7f7a69ffb700 1 -- 172.21.15.78:0/853675475 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f7a7c10df90 con 0x7f7a6402fd70 2023-12-21T03:39:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Self-signed certificate created 2023-12-21T03:39:32.360 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.358+0000 7f7a822e5700 1 -- 172.21.15.78:0/853675475 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a6402fd70 msgr2=0x7f7a64032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:32.360 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.358+0000 7f7a822e5700 1 --2- 172.21.15.78:0/853675475 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a6402fd70 0x7f7a64032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7a6c0096a0 tx=0x7f7a6c006b40).stop 2023-12-21T03:39:32.360 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.358+0000 7f7a822e5700 1 -- 172.21.15.78:0/853675475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 msgr2=0x7f7a7c10cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:32.360 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.358+0000 7f7a822e5700 1 --2- 172.21.15.78:0/853675475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c10cb50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f7a78006800 tx=0x7f7a78006ee0).stop 2023-12-21T03:39:32.361 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.358+0000 7f7a822e5700 1 -- 172.21.15.78:0/853675475 shutdown_connections 2023-12-21T03:39:32.361 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.358+0000 7f7a822e5700 1 --2- 172.21.15.78:0/853675475 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a6402fd70 0x7f7a64032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:32.361 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.358+0000 7f7a822e5700 1 --2- 172.21.15.78:0/853675475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0724d0 0x7f7a7c10cb50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:32.361 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.358+0000 7f7a822e5700 1 -- 172.21.15.78:0/853675475 >> 172.21.15.78:0/853675475 conn(0x7f7a7c0f7380 msgr2=0x7f7a7c0f8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:32.361 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.358+0000 7f7a822e5700 1 -- 172.21.15.78:0/853675475 shutdown_connections 2023-12-21T03:39:32.361 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:32.358+0000 7f7a822e5700 1 -- 172.21.15.78:0/853675475 wait complete. 2023-12-21T03:39:32.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:31.401726+0000 mgr.smithi078.zidbgq (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-21T03:39:32.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:31.402356+0000 mon.smithi078 (mon.0) 105 : cluster [DBG] mgrmap e15: smithi078.zidbgq(active, since 1.05796s) 2023-12-21T03:39:32.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:31.408227+0000 mgr.smithi078.zidbgq (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-21T03:39:32.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:31.463541+0000 mon.smithi078 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:32.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:32.350636+0000 mon.smithi078 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:32.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:32.357469+0000 mon.smithi078 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:32.873 INFO:teuthology.orchestra.run.smithi078.stderr:Creating initial admin user... 2023-12-21T03:39:32.873 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpd7wqdyl3:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-12-21T03:39:33.240 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.233+0000 7f1ef96c2700 1 Processor -- start 2023-12-21T03:39:33.240 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.233+0000 7f1ef96c2700 1 -- start start 2023-12-21T03:39:33.240 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.233+0000 7f1ef96c2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:33.240 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.233+0000 7f1ef96c2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef40fb490 con 0x7f1ef40fab80 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.234+0000 7f1ef2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.234+0000 7f1ef2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55144/0 (socket says 172.21.15.78:55144) 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.234+0000 7f1ef2ffd700 1 -- 172.21.15.78:0/3430234036 learned_addr learned my addr 172.21.15.78:0/3430234036 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.234+0000 7f1ef2ffd700 1 -- 172.21.15.78:0/3430234036 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef40fb5d0 con 0x7f1ef40fab80 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.234+0000 7f1ef2ffd700 1 --2- 172.21.15.78:0/3430234036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef40fafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1ee800b5f0 tx=0x7f1ee8006ee0).ready entity=mon.0 client_cookie=269e288898763bde server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.234+0000 7f1ef1ffb700 1 -- 172.21.15.78:0/3430234036 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ee800d600 con 0x7f1ef40fab80 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.234+0000 7f1ef1ffb700 1 -- 172.21.15.78:0/3430234036 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ee800dc10 con 0x7f1ef40fab80 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.234+0000 7f1ef1ffb700 1 -- 172.21.15.78:0/3430234036 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ee8011cd0 con 0x7f1ef40fab80 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.235+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3430234036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 msgr2=0x7f1ef40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.235+0000 7f1ef96c2700 1 --2- 172.21.15.78:0/3430234036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef40fafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1ee800b5f0 tx=0x7f1ee8006ee0).stop 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.235+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3430234036 shutdown_connections 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.235+0000 7f1ef96c2700 1 --2- 172.21.15.78:0/3430234036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef40fafa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:33.241 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.235+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3430234036 >> 172.21.15.78:0/3430234036 conn(0x7f1ef40f63c0 msgr2=0x7f1ef40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.235+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3430234036 shutdown_connections 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.235+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3430234036 wait complete. 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.235+0000 7f1ef96c2700 1 Processor -- start 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.235+0000 7f1ef96c2700 1 -- start start 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1ef96c2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef41099c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1ef96c2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef4109eb0 con 0x7f1ef40fab80 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1ef2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef41099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1ef2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef41099c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55158/0 (socket says 172.21.15.78:55158) 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1ef2ffd700 1 -- 172.21.15.78:0/3526317136 learned_addr learned my addr 172.21.15.78:0/3526317136 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1ef2ffd700 1 -- 172.21.15.78:0/3526317136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee800b200 con 0x7f1ef40fab80 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1ef2ffd700 1 --2- 172.21.15.78:0/3526317136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef41099c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1ee800e040 tx=0x7f1ee8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:33.242 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1edb7fe700 1 -- 172.21.15.78:0/3526317136 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ee801b410 con 0x7f1ef40fab80 2023-12-21T03:39:33.243 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3526317136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef410a0b0 con 0x7f1ef40fab80 2023-12-21T03:39:33.243 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1edb7fe700 1 -- 172.21.15.78:0/3526317136 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ee801ba20 con 0x7f1ef40fab80 2023-12-21T03:39:33.243 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1edb7fe700 1 -- 172.21.15.78:0/3526317136 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ee801a910 con 0x7f1ef40fab80 2023-12-21T03:39:33.243 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.236+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3526317136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef410a4c0 con 0x7f1ef40fab80 2023-12-21T03:39:33.243 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.237+0000 7f1edb7fe700 1 -- 172.21.15.78:0/3526317136 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f1ee800f070 con 0x7f1ef40fab80 2023-12-21T03:39:33.243 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.237+0000 7f1edb7fe700 1 --2- 172.21.15.78:0/3526317136 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1ee002fd50 0x7f1ee0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:33.243 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.237+0000 7f1edb7fe700 1 -- 172.21.15.78:0/3526317136 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ee80411c0 con 0x7f1ef40fab80 2023-12-21T03:39:33.243 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.238+0000 7f1ef27fc700 1 --2- 172.21.15.78:0/3526317136 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1ee002fd50 0x7f1ee0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:33.243 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.238+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3526317136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef4059100 con 0x7f1ef40fab80 2023-12-21T03:39:33.243 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.238+0000 7f1ef27fc700 1 --2- 172.21.15.78:0/3526317136 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1ee002fd50 0x7f1ee0032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f1ee40096a0 tx=0x7f1ee4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:33.244 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.242+0000 7f1edb7fe700 1 -- 172.21.15.78:0/3526317136 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ee8016070 con 0x7f1ef40fab80 2023-12-21T03:39:33.352 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.350+0000 7f1edb7fe700 1 -- 172.21.15.78:0/3526317136 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f1ee8024930 con 0x7f1ef40fab80 2023-12-21T03:39:33.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.411+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3526317136 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1ef41037c0 con 0x7f1ee002fd50 2023-12-21T03:39:33.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.630+0000 7f1edb7fe700 1 -- 172.21.15.78:0/3526317136 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1ef41037c0 con 0x7f1ee002fd50 2023-12-21T03:39:33.634 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$tOmCeo/7sESFefM3N422X.tjQT.1E5AB/ic9AqCpjiErvhHUNhDFe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1703129973, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-21T03:39:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.631+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3526317136 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1ee002fd50 msgr2=0x7f1ee0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.632+0000 7f1ef96c2700 1 --2- 172.21.15.78:0/3526317136 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1ee002fd50 0x7f1ee0032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f1ee40096a0 tx=0x7f1ee4006b40).stop 2023-12-21T03:39:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.632+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3526317136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 msgr2=0x7f1ef41099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.632+0000 7f1ef96c2700 1 --2- 172.21.15.78:0/3526317136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef41099c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1ee800e040 tx=0x7f1ee8006ee0).stop 2023-12-21T03:39:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.632+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3526317136 shutdown_connections 2023-12-21T03:39:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.632+0000 7f1ef96c2700 1 --2- 172.21.15.78:0/3526317136 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1ee002fd50 0x7f1ee0032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.632+0000 7f1ef96c2700 1 --2- 172.21.15.78:0/3526317136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ef40fab80 0x7f1ef41099c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.632+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3526317136 >> 172.21.15.78:0/3526317136 conn(0x7f1ef40f63c0 msgr2=0x7f1ef40f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.632+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3526317136 shutdown_connections 2023-12-21T03:39:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:33.632+0000 7f1ef96c2700 1 -- 172.21.15.78:0/3526317136 wait complete. 2023-12-21T03:39:33.659 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:32.298267+0000 mgr.smithi078.zidbgq (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:33.660 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:33.352428+0000 mon.smithi078 (mon.0) 109 : cluster [DBG] mgrmap e16: smithi078.zidbgq(active, since 3s) 2023-12-21T03:39:34.006 INFO:teuthology.orchestra.run.smithi078.stderr:Fetching dashboard port number... 2023-12-21T03:39:34.006 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpstpzzzni:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpi71489wx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-12-21T03:39:34.405 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.402+0000 7fb14294a700 1 Processor -- start 2023-12-21T03:39:34.405 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.402+0000 7fb14294a700 1 -- start start 2023-12-21T03:39:34.405 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.403+0000 7fb14294a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c0fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.403+0000 7fb14294a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c0fc5b0 con 0x7fb13c0fbca0 2023-12-21T03:39:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.403+0000 7fb13bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c0fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.403+0000 7fb13bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c0fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55164/0 (socket says 172.21.15.78:55164) 2023-12-21T03:39:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.403+0000 7fb13bfff700 1 -- 172.21.15.78:0/1883004964 learned_addr learned my addr 172.21.15.78:0/1883004964 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.403+0000 7fb13bfff700 1 -- 172.21.15.78:0/1883004964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c0fc6f0 con 0x7fb13c0fbca0 2023-12-21T03:39:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.403+0000 7fb13bfff700 1 --2- 172.21.15.78:0/1883004964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c0fc0c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb12c00b5f0 tx=0x7fb12c006ee0).ready entity=mon.0 client_cookie=16046ef1a20dfe8b server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.404+0000 7fb13affd700 1 -- 172.21.15.78:0/1883004964 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c00d600 con 0x7fb13c0fbca0 2023-12-21T03:39:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.404+0000 7fb13affd700 1 -- 172.21.15.78:0/1883004964 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb12c00dc10 con 0x7fb13c0fbca0 2023-12-21T03:39:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.404+0000 7fb13affd700 1 -- 172.21.15.78:0/1883004964 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c011cd0 con 0x7fb13c0fbca0 2023-12-21T03:39:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.405+0000 7fb14294a700 1 -- 172.21.15.78:0/1883004964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 msgr2=0x7fb13c0fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.405+0000 7fb14294a700 1 --2- 172.21.15.78:0/1883004964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c0fc0c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb12c00b5f0 tx=0x7fb12c006ee0).stop 2023-12-21T03:39:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.405+0000 7fb14294a700 1 -- 172.21.15.78:0/1883004964 shutdown_connections 2023-12-21T03:39:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.405+0000 7fb14294a700 1 --2- 172.21.15.78:0/1883004964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c0fc0c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.405+0000 7fb14294a700 1 -- 172.21.15.78:0/1883004964 >> 172.21.15.78:0/1883004964 conn(0x7fb13c0f74c0 msgr2=0x7fb13c0f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.405+0000 7fb14294a700 1 -- 172.21.15.78:0/1883004964 shutdown_connections 2023-12-21T03:39:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.405+0000 7fb14294a700 1 -- 172.21.15.78:0/1883004964 wait complete. 2023-12-21T03:39:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.405+0000 7fb14294a700 1 Processor -- start 2023-12-21T03:39:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb14294a700 1 -- start start 2023-12-21T03:39:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb14294a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb14294a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c10aef0 con 0x7fb13c0fbca0 2023-12-21T03:39:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb13bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb13bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55178/0 (socket says 172.21.15.78:55178) 2023-12-21T03:39:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb13bfff700 1 -- 172.21.15.78:0/395109324 learned_addr learned my addr 172.21.15.78:0/395109324 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb13bfff700 1 -- 172.21.15.78:0/395109324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c00b200 con 0x7fb13c0fbca0 2023-12-21T03:39:34.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb13bfff700 1 --2- 172.21.15.78:0/395109324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c10aa00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fb12c00e040 tx=0x7fb12c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:34.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb138ff9700 1 -- 172.21.15.78:0/395109324 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c01b410 con 0x7fb13c0fbca0 2023-12-21T03:39:34.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb14294a700 1 -- 172.21.15.78:0/395109324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c10b0f0 con 0x7fb13c0fbca0 2023-12-21T03:39:34.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb138ff9700 1 -- 172.21.15.78:0/395109324 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb12c01ba20 con 0x7fb13c0fbca0 2023-12-21T03:39:34.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb138ff9700 1 -- 172.21.15.78:0/395109324 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c01a910 con 0x7fb13c0fbca0 2023-12-21T03:39:34.410 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.406+0000 7fb14294a700 1 -- 172.21.15.78:0/395109324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c10b500 con 0x7fb13c0fbca0 2023-12-21T03:39:34.411 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.407+0000 7fb138ff9700 1 -- 172.21.15.78:0/395109324 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fb12c00f070 con 0x7fb13c0fbca0 2023-12-21T03:39:34.411 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.407+0000 7fb138ff9700 1 --2- 172.21.15.78:0/395109324 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb12802feb0 0x7fb128032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:34.411 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.407+0000 7fb138ff9700 1 -- 172.21.15.78:0/395109324 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb12c042220 con 0x7fb13c0fbca0 2023-12-21T03:39:34.411 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.407+0000 7fb13b7fe700 1 --2- 172.21.15.78:0/395109324 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb12802feb0 0x7fb128032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:34.411 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.407+0000 7fb14294a700 1 -- 172.21.15.78:0/395109324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c059100 con 0x7fb13c0fbca0 2023-12-21T03:39:34.411 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.408+0000 7fb13b7fe700 1 --2- 172.21.15.78:0/395109324 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb12802feb0 0x7fb128032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb1300096a0 tx=0x7fb130006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:34.413 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.412+0000 7fb138ff9700 1 -- 172.21.15.78:0/395109324 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb12c010be0 con 0x7fb13c0fbca0 2023-12-21T03:39:34.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.560+0000 7fb14294a700 1 -- 172.21.15.78:0/395109324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb13c0fda20 con 0x7fb13c0fbca0 2023-12-21T03:39:34.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.560+0000 7fb138ff9700 1 -- 172.21.15.78:0/395109324 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb12c0107d0 con 0x7fb13c0fbca0 2023-12-21T03:39:34.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 8443 2023-12-21T03:39:34.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.562+0000 7fb14294a700 1 -- 172.21.15.78:0/395109324 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb12802feb0 msgr2=0x7fb128032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:34.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.562+0000 7fb14294a700 1 --2- 172.21.15.78:0/395109324 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb12802feb0 0x7fb128032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb1300096a0 tx=0x7fb130006b40).stop 2023-12-21T03:39:34.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.562+0000 7fb14294a700 1 -- 172.21.15.78:0/395109324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 msgr2=0x7fb13c10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:34.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.562+0000 7fb14294a700 1 --2- 172.21.15.78:0/395109324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c10aa00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fb12c00e040 tx=0x7fb12c006ee0).stop 2023-12-21T03:39:34.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.562+0000 7fb14294a700 1 -- 172.21.15.78:0/395109324 shutdown_connections 2023-12-21T03:39:34.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.562+0000 7fb14294a700 1 --2- 172.21.15.78:0/395109324 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb12802feb0 0x7fb128032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:34.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.562+0000 7fb14294a700 1 --2- 172.21.15.78:0/395109324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c0fbca0 0x7fb13c10aa00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:34.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.562+0000 7fb14294a700 1 -- 172.21.15.78:0/395109324 >> 172.21.15.78:0/395109324 conn(0x7fb13c0f74c0 msgr2=0x7fb13c0f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:34.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.562+0000 7fb14294a700 1 -- 172.21.15.78:0/395109324 shutdown_connections 2023-12-21T03:39:34.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-12-21T03:39:34.562+0000 7fb14294a700 1 -- 172.21.15.78:0/395109324 wait complete. 2023-12-21T03:39:34.721 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:34 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:33.413182+0000 mgr.smithi078.zidbgq (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:39:34.721 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:34 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:33.631184+0000 mon.smithi078 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:34.915 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-21T03:39:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: disabled 2023-12-21T03:39:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active firewalld.service 2023-12-21T03:39:34.930 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-12-21T03:39:34.930 INFO:teuthology.orchestra.run.smithi078.stderr:firewalld.service is not enabled 2023-12-21T03:39:34.930 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-21T03:39:34.933 INFO:teuthology.orchestra.run.smithi078.stderr:Ceph Dashboard is now available at: 2023-12-21T03:39:34.933 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-12-21T03:39:34.933 INFO:teuthology.orchestra.run.smithi078.stderr: URL: https://smithi078.front.sepia.ceph.com:8443/ 2023-12-21T03:39:34.933 INFO:teuthology.orchestra.run.smithi078.stderr: User: admin 2023-12-21T03:39:34.933 INFO:teuthology.orchestra.run.smithi078.stderr: Password: 3ohhxxlscy 2023-12-21T03:39:34.933 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-12-21T03:39:34.934 INFO:teuthology.orchestra.run.smithi078.stderr:You can access the Ceph CLI with: 2023-12-21T03:39:34.934 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-12-21T03:39:34.934 INFO:teuthology.orchestra.run.smithi078.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-21T03:39:34.934 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-12-21T03:39:34.934 INFO:teuthology.orchestra.run.smithi078.stderr:Please consider enabling telemetry to help improve Ceph: 2023-12-21T03:39:34.934 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-12-21T03:39:34.934 INFO:teuthology.orchestra.run.smithi078.stderr: ceph telemetry on 2023-12-21T03:39:34.935 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-12-21T03:39:34.935 INFO:teuthology.orchestra.run.smithi078.stderr:For more information see: 2023-12-21T03:39:34.935 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-12-21T03:39:34.935 INFO:teuthology.orchestra.run.smithi078.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-12-21T03:39:34.935 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-12-21T03:39:34.935 INFO:teuthology.orchestra.run.smithi078.stderr:Bootstrap complete. 2023-12-21T03:39:34.935 INFO:teuthology.orchestra.run.smithi078.stderr:Releasing lock 139970846759400 on /run/cephadm/5ef0788c-9fb2-11ee-95a5-87774f69a715.lock 2023-12-21T03:39:34.935 INFO:teuthology.orchestra.run.smithi078.stderr:Lock 139970846759400 released on /run/cephadm/5ef0788c-9fb2-11ee-95a5-87774f69a715.lock 2023-12-21T03:39:34.967 INFO:tasks.cephadm:Fetching config... 2023-12-21T03:39:34.967 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:39:34.967 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-21T03:39:35.025 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-21T03:39:35.025 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:39:35.025 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-21T03:39:35.083 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-21T03:39:35.083 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:39:35.083 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/keyring of=/dev/stdout 2023-12-21T03:39:35.152 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-21T03:39:35.152 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:39:35.152 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-21T03:39:35.210 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-21T03:39:35.210 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4utIP2qTeLRtf9My0CWyNka/Jf0Vg/RAZiPO4lQi3qV/SH51SSFcPBoAAcg8OP/2mxyaEGlQ0Si1o5URr1eUghaUr3ZoPmokScOH6J/lml/1uVP7t2F4mwq5989q7XAWIQMe1PhNEkUZCkQQBt5wvSsP1tYMmBsod2r0CBhZmSooQY3GWt9npmh6HFmgWzK8W0UzwWB83SdTjZkoDhWtJbbkHjh0MntLbU+I17PuqU5yQKeruOafLHSWi/856DpEC1S3zGE9Vs1qfes9h2hy69/FahoYi93f/U5yKyYmL/IcOTw4hkfx5IYVDCsq8Q0B74AMtYpI9vpYOxMnz2Q813UMN/FW68F9VgIg7owGRN2QGZOoqAr5S+PSylj7JLpypzKaW0SXUoOg1Y4JFfqFWqDCDh+04kB8cgvmCNEuISI8HO/dAaIL/SIb9Z5HqgJpHg7XpP5Lp+2P4Yxp6K1Tu50EkcyYatJx1AubFhZ8Z6i/eikEzJlkoja+ny8hAp7M= ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-21T03:39:35.287 INFO:teuthology.orchestra.run.smithi078.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4utIP2qTeLRtf9My0CWyNka/Jf0Vg/RAZiPO4lQi3qV/SH51SSFcPBoAAcg8OP/2mxyaEGlQ0Si1o5URr1eUghaUr3ZoPmokScOH6J/lml/1uVP7t2F4mwq5989q7XAWIQMe1PhNEkUZCkQQBt5wvSsP1tYMmBsod2r0CBhZmSooQY3GWt9npmh6HFmgWzK8W0UzwWB83SdTjZkoDhWtJbbkHjh0MntLbU+I17PuqU5yQKeruOafLHSWi/856DpEC1S3zGE9Vs1qfes9h2hy69/FahoYi93f/U5yKyYmL/IcOTw4hkfx5IYVDCsq8Q0B74AMtYpI9vpYOxMnz2Q813UMN/FW68F9VgIg7owGRN2QGZOoqAr5S+PSylj7JLpypzKaW0SXUoOg1Y4JFfqFWqDCDh+04kB8cgvmCNEuISI8HO/dAaIL/SIb9Z5HqgJpHg7XpP5Lp+2P4Yxp6K1Tu50EkcyYatJx1AubFhZ8Z6i/eikEzJlkoja+ny8hAp7M= ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:39:35.298 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4utIP2qTeLRtf9My0CWyNka/Jf0Vg/RAZiPO4lQi3qV/SH51SSFcPBoAAcg8OP/2mxyaEGlQ0Si1o5URr1eUghaUr3ZoPmokScOH6J/lml/1uVP7t2F4mwq5989q7XAWIQMe1PhNEkUZCkQQBt5wvSsP1tYMmBsod2r0CBhZmSooQY3GWt9npmh6HFmgWzK8W0UzwWB83SdTjZkoDhWtJbbkHjh0MntLbU+I17PuqU5yQKeruOafLHSWi/856DpEC1S3zGE9Vs1qfes9h2hy69/FahoYi93f/U5yKyYmL/IcOTw4hkfx5IYVDCsq8Q0B74AMtYpI9vpYOxMnz2Q813UMN/FW68F9VgIg7owGRN2QGZOoqAr5S+PSylj7JLpypzKaW0SXUoOg1Y4JFfqFWqDCDh+04kB8cgvmCNEuISI8HO/dAaIL/SIb9Z5HqgJpHg7XpP5Lp+2P4Yxp6K1Tu50EkcyYatJx1AubFhZ8Z6i/eikEzJlkoja+ny8hAp7M= ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-21T03:39:35.344 INFO:teuthology.orchestra.run.smithi148.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4utIP2qTeLRtf9My0CWyNka/Jf0Vg/RAZiPO4lQi3qV/SH51SSFcPBoAAcg8OP/2mxyaEGlQ0Si1o5URr1eUghaUr3ZoPmokScOH6J/lml/1uVP7t2F4mwq5989q7XAWIQMe1PhNEkUZCkQQBt5wvSsP1tYMmBsod2r0CBhZmSooQY3GWt9npmh6HFmgWzK8W0UzwWB83SdTjZkoDhWtJbbkHjh0MntLbU+I17PuqU5yQKeruOafLHSWi/856DpEC1S3zGE9Vs1qfes9h2hy69/FahoYi93f/U5yKyYmL/IcOTw4hkfx5IYVDCsq8Q0B74AMtYpI9vpYOxMnz2Q813UMN/FW68F9VgIg7owGRN2QGZOoqAr5S+PSylj7JLpypzKaW0SXUoOg1Y4JFfqFWqDCDh+04kB8cgvmCNEuISI8HO/dAaIL/SIb9Z5HqgJpHg7XpP5Lp+2P4Yxp6K1Tu50EkcyYatJx1AubFhZ8Z6i/eikEzJlkoja+ny8hAp7M= ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:39:35.356 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-21T03:39:35.679 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:34.561829+0000 mon.smithi078 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.78:0/395109324' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-21T03:39:37.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.033+0000 7faae359e700 1 -- 172.21.15.78:0/77663669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faae40f51a0 msgr2=0x7faae40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:37.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.033+0000 7faae359e700 1 --2- 172.21.15.78:0/77663669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faae40f51a0 0x7faae40f55c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7faad4009fb0 tx=0x7faad4009b30).stop 2023-12-21T03:39:37.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.033+0000 7faae359e700 1 -- 172.21.15.78:0/77663669 shutdown_connections 2023-12-21T03:39:37.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.033+0000 7faae359e700 1 --2- 172.21.15.78:0/77663669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faae40f51a0 0x7faae40f55c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:37.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.033+0000 7faae359e700 1 -- 172.21.15.78:0/77663669 >> 172.21.15.78:0/77663669 conn(0x7faae40f09a0 msgr2=0x7faae40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:37.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.033+0000 7faae359e700 1 -- 172.21.15.78:0/77663669 shutdown_connections 2023-12-21T03:39:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.034+0000 7faae359e700 1 -- 172.21.15.78:0/77663669 wait complete. 2023-12-21T03:39:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.034+0000 7faae359e700 1 Processor -- start 2023-12-21T03:39:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.034+0000 7faae359e700 1 -- start start 2023-12-21T03:39:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.034+0000 7faae359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faae40f51a0 0x7faae41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.034+0000 7faae359e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae4060220 con 0x7faae40f51a0 2023-12-21T03:39:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.035+0000 7faae259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faae40f51a0 0x7faae41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.035+0000 7faae259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faae40f51a0 0x7faae41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55194/0 (socket says 172.21.15.78:55194) 2023-12-21T03:39:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.035+0000 7faae259c700 1 -- 172.21.15.78:0/2322589890 learned_addr learned my addr 172.21.15.78:0/2322589890 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:39:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.035+0000 7faae259c700 1 -- 172.21.15.78:0/2322589890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad4005040 con 0x7faae40f51a0 2023-12-21T03:39:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.035+0000 7faae259c700 1 --2- 172.21.15.78:0/2322589890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faae40f51a0 0x7faae41091d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7faad4013510 tx=0x7faad400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.035+0000 7faad2ffd700 1 -- 172.21.15.78:0/2322589890 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faad4009320 con 0x7faae40f51a0 2023-12-21T03:39:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.035+0000 7faad2ffd700 1 -- 172.21.15.78:0/2322589890 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faad4013f00 con 0x7faae40f51a0 2023-12-21T03:39:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.035+0000 7faad2ffd700 1 -- 172.21.15.78:0/2322589890 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faad4014dc0 con 0x7faae40f51a0 2023-12-21T03:39:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.035+0000 7faae359e700 1 -- 172.21.15.78:0/2322589890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae4060420 con 0x7faae40f51a0 2023-12-21T03:39:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.035+0000 7faae359e700 1 -- 172.21.15.78:0/2322589890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae4109880 con 0x7faae40f51a0 2023-12-21T03:39:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.036+0000 7faad2ffd700 1 -- 172.21.15.78:0/2322589890 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faad401d3d0 con 0x7faae40f51a0 2023-12-21T03:39:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.036+0000 7faae359e700 1 -- 172.21.15.78:0/2322589890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae4059100 con 0x7faae40f51a0 2023-12-21T03:39:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.036+0000 7faad2ffd700 1 --2- 172.21.15.78:0/2322589890 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faacc02ff60 0x7faacc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.036+0000 7faad2ffd700 1 -- 172.21.15.78:0/2322589890 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faad4041370 con 0x7faae40f51a0 2023-12-21T03:39:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.037+0000 7faae1d9b700 1 --2- 172.21.15.78:0/2322589890 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faacc02ff60 0x7faacc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:37.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.037+0000 7faae1d9b700 1 --2- 172.21.15.78:0/2322589890 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faacc02ff60 0x7faacc032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7faad80096a0 tx=0x7faad8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:37.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.041+0000 7faad2ffd700 1 -- 172.21.15.78:0/2322589890 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faad40126f0 con 0x7faae40f51a0 2023-12-21T03:39:37.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.199+0000 7faae359e700 1 -- 172.21.15.78:0/2322589890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7faae4045ad0 con 0x7faae40f51a0 2023-12-21T03:39:37.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.202+0000 7faad2ffd700 1 -- 172.21.15.78:0/2322589890 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7faad4012330 con 0x7faae40f51a0 2023-12-21T03:39:37.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.204+0000 7faae359e700 1 -- 172.21.15.78:0/2322589890 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faacc02ff60 msgr2=0x7faacc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:37.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.204+0000 7faae359e700 1 --2- 172.21.15.78:0/2322589890 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faacc02ff60 0x7faacc032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7faad80096a0 tx=0x7faad8006b40).stop 2023-12-21T03:39:37.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.204+0000 7faae359e700 1 -- 172.21.15.78:0/2322589890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faae40f51a0 msgr2=0x7faae41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:37.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.204+0000 7faae359e700 1 --2- 172.21.15.78:0/2322589890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faae40f51a0 0x7faae41091d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7faad4013510 tx=0x7faad400b040).stop 2023-12-21T03:39:37.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.206+0000 7faae359e700 1 -- 172.21.15.78:0/2322589890 shutdown_connections 2023-12-21T03:39:37.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.206+0000 7faae359e700 1 --2- 172.21.15.78:0/2322589890 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faacc02ff60 0x7faacc032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:37.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.206+0000 7faae359e700 1 --2- 172.21.15.78:0/2322589890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faae40f51a0 0x7faae41091d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:37.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.206+0000 7faae359e700 1 -- 172.21.15.78:0/2322589890 >> 172.21.15.78:0/2322589890 conn(0x7faae40f09a0 msgr2=0x7faae40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:37.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.206+0000 7faae359e700 1 -- 172.21.15.78:0/2322589890 shutdown_connections 2023-12-21T03:39:37.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:39:37.206+0000 7faae359e700 1 -- 172.21.15.78:0/2322589890 wait complete. 2023-12-21T03:39:37.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:36.406976+0000 mon.smithi078 (mon.0) 112 : cluster [DBG] mgrmap e17: smithi078.zidbgq(active, since 6s) 2023-12-21T03:39:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:37.013979+0000 mon.smithi078 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:37.203736+0000 mon.smithi078 (mon.0) 114 : audit [INF] from='client.? 172.21.15.78:0/2322589890' entity='client.admin' 2023-12-21T03:39:37.631 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi148 2023-12-21T03:39:37.632 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:39:37.632 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/etc/ceph/ceph.conf 2023-12-21T03:39:37.654 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:39:37.654 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-21T03:39:37.715 INFO:tasks.cephadm:Adding host smithi148 to orchestrator... 2023-12-21T03:39:37.715 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph orch host add smithi148 2023-12-21T03:39:38.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:37.435107+0000 mon.smithi078 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:38.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:37.441587+0000 mgr.smithi078.zidbgq (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi078 on smithi078 2023-12-21T03:39:49.807 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:48.725573+0000 mon.smithi078 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:49.807 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:48.726597+0000 mon.smithi078 (mon.0) 117 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi078", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-21T03:39:49.808 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:48.728217+0000 mon.smithi078 (mon.0) 118 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi078", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-21T03:39:49.808 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:48.728747+0000 mon.smithi078 (mon.0) 119 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:39:49.808 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:48.729226+0000 mgr.smithi078.zidbgq (mgr.14162) 6 : cephadm [INF] Deploying daemon crash.smithi078 on smithi078 2023-12-21T03:39:51.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:50.350791+0000 mon.smithi078 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:51.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:50.403364+0000 mgr.smithi078.zidbgq (mgr.14162) 7 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:39:51.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:50.462270+0000 mon.smithi078 (mon.0) 121 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:51.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:50.464058+0000 mon.smithi078 (mon.0) 122 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:51.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:50.464433+0000 mon.smithi078 (mon.0) 123 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-21T03:39:51.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:50.467856+0000 mon.smithi078 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:39:52.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:50.464801+0000 mgr.smithi078.zidbgq (mgr.14162) 8 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-21T03:39:52.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:50.470759+0000 mgr.smithi078.zidbgq (mgr.14162) 9 : cephadm [INF] Deploying daemon grafana.smithi078 on smithi078 2023-12-21T03:39:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:52.403726+0000 mgr.smithi078.zidbgq (mgr.14162) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:39:55.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:54.404121+0000 mgr.smithi078.zidbgq (mgr.14162) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:39:56.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:56.404355+0000 mgr.smithi078.zidbgq (mgr.14162) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:39:58.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:39:58 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:39:58.404605+0000 mgr.smithi078.zidbgq (mgr.14162) 13 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:39:59.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.134+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1967969485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2b40f62a0 msgr2=0x7fb2b40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:59.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.134+0000 7fb2b3fff700 1 --2- 172.21.15.148:0/1967969485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2b40f62a0 0x7fb2b40f66c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb2ac009fb0 tx=0x7fb2ac009b30).stop 2023-12-21T03:39:59.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.134+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1967969485 shutdown_connections 2023-12-21T03:39:59.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.134+0000 7fb2b3fff700 1 --2- 172.21.15.148:0/1967969485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2b40f62a0 0x7fb2b40f66c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:59.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.134+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1967969485 >> 172.21.15.148:0/1967969485 conn(0x7fb2b40f1ac0 msgr2=0x7fb2b40f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.134+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1967969485 shutdown_connections 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.134+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1967969485 wait complete. 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.134+0000 7fb2b3fff700 1 Processor -- start 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.134+0000 7fb2b3fff700 1 -- start start 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2b3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2b40f62a0 0x7fb2b41086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2b3fff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b4108ba0 con 0x7fb2b40f62a0 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2b2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2b40f62a0 0x7fb2b41086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2b2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2b40f62a0 0x7fb2b41086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:50150/0 (socket says 172.21.15.148:50150) 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2b2ffd700 1 -- 172.21.15.148:0/1089261737 learned_addr learned my addr 172.21.15.148:0/1089261737 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2b2ffd700 1 -- 172.21.15.148:0/1089261737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ac005040 con 0x7fb2b40f62a0 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2b2ffd700 1 --2- 172.21.15.148:0/1089261737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2b40f62a0 0x7fb2b41086b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fb2ac009830 tx=0x7fb2ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2ab7fe700 1 -- 172.21.15.148:0/1089261737 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2ac009320 con 0x7fb2b40f62a0 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1089261737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b4108da0 con 0x7fb2b40f62a0 2023-12-21T03:39:59.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2ab7fe700 1 -- 172.21.15.148:0/1089261737 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2ac0136c0 con 0x7fb2b40f62a0 2023-12-21T03:39:59.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2ab7fe700 1 -- 172.21.15.148:0/1089261737 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2ac01ee90 con 0x7fb2b40f62a0 2023-12-21T03:39:59.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.135+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1089261737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b4109230 con 0x7fb2b40f62a0 2023-12-21T03:39:59.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.136+0000 7fb2ab7fe700 1 -- 172.21.15.148:0/1089261737 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2ac014440 con 0x7fb2b40f62a0 2023-12-21T03:39:59.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.136+0000 7fb2ab7fe700 1 --2- 172.21.15.148:0/1089261737 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb2a002ff10 0x7fb2a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:39:59.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.136+0000 7fb2ab7fe700 1 -- 172.21.15.148:0/1089261737 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2ac040f30 con 0x7fb2b40f62a0 2023-12-21T03:39:59.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.136+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1089261737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b4045ad0 con 0x7fb2b40f62a0 2023-12-21T03:39:59.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.137+0000 7fb2b27fc700 1 --2- 172.21.15.148:0/1089261737 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb2a002ff10 0x7fb2a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:39:59.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.137+0000 7fb2b27fc700 1 --2- 172.21.15.148:0/1089261737 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb2a002ff10 0x7fb2a00323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb29c0096a0 tx=0x7fb29c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:39:59.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.141+0000 7fb2ab7fe700 1 -- 172.21.15.148:0/1089261737 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2ac0273f0 con 0x7fb2b40f62a0 2023-12-21T03:39:59.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.312+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1089261737 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi148", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2b41022e0 con 0x7fb2a002ff10 2023-12-21T03:39:59.902 INFO:teuthology.orchestra.run.smithi148.stdout:Added host 'smithi148' 2023-12-21T03:39:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.900+0000 7fb2ab7fe700 1 -- 172.21.15.148:0/1089261737 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fb2b41022e0 con 0x7fb2a002ff10 2023-12-21T03:39:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.903+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1089261737 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb2a002ff10 msgr2=0x7fb2a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.903+0000 7fb2b3fff700 1 --2- 172.21.15.148:0/1089261737 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb2a002ff10 0x7fb2a00323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb29c0096a0 tx=0x7fb29c006b40).stop 2023-12-21T03:39:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.903+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1089261737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2b40f62a0 msgr2=0x7fb2b41086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:39:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.903+0000 7fb2b3fff700 1 --2- 172.21.15.148:0/1089261737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2b40f62a0 0x7fb2b41086b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fb2ac009830 tx=0x7fb2ac00b040).stop 2023-12-21T03:39:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.903+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1089261737 shutdown_connections 2023-12-21T03:39:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.903+0000 7fb2b3fff700 1 --2- 172.21.15.148:0/1089261737 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb2a002ff10 0x7fb2a00323d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.903+0000 7fb2b3fff700 1 --2- 172.21.15.148:0/1089261737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2b40f62a0 0x7fb2b41086b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:39:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.903+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1089261737 >> 172.21.15.148:0/1089261737 conn(0x7fb2b40f1ac0 msgr2=0x7fb2b40f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:39:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.903+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1089261737 shutdown_connections 2023-12-21T03:39:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:39:59.903+0000 7fb2b3fff700 1 -- 172.21.15.148:0/1089261737 wait complete. 2023-12-21T03:40:00.235 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph orch host ls --format=json 2023-12-21T03:40:00.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:59.314005+0000 mgr.smithi078.zidbgq (mgr.14162) 14 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi148", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:40:00.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:39:59.900160+0000 mon.smithi078 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:40:01.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:39:59.900533+0000 mgr.smithi078.zidbgq (mgr.14162) 15 : cephadm [INF] Added host smithi148 2023-12-21T03:40:01.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:00.404827+0000 mgr.smithi078.zidbgq (mgr.14162) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:02.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.424+0000 7f30ebe19700 1 -- 172.21.15.148:0/1660642608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e40f5410 msgr2=0x7f30e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:02.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.424+0000 7f30ebe19700 1 --2- 172.21.15.148:0/1660642608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e40f5410 0x7f30e40f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f30d4009fb0 tx=0x7f30d4009b30).stop 2023-12-21T03:40:02.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.424+0000 7f30ebe19700 1 -- 172.21.15.148:0/1660642608 shutdown_connections 2023-12-21T03:40:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.424+0000 7f30ebe19700 1 --2- 172.21.15.148:0/1660642608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e40f5410 0x7f30e40f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.424+0000 7f30ebe19700 1 -- 172.21.15.148:0/1660642608 >> 172.21.15.148:0/1660642608 conn(0x7f30e40f0b70 msgr2=0x7f30e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.424+0000 7f30ebe19700 1 -- 172.21.15.148:0/1660642608 shutdown_connections 2023-12-21T03:40:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.424+0000 7f30ebe19700 1 -- 172.21.15.148:0/1660642608 wait complete. 2023-12-21T03:40:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.425+0000 7f30ebe19700 1 Processor -- start 2023-12-21T03:40:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.425+0000 7f30ebe19700 1 -- start start 2023-12-21T03:40:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.425+0000 7f30ebe19700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e40f5410 0x7f30e4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.425+0000 7f30ebe19700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e4060240 con 0x7f30e40f5410 2023-12-21T03:40:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.425+0000 7f30e9bb5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e40f5410 0x7f30e4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.425+0000 7f30e9bb5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e40f5410 0x7f30e4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:59968/0 (socket says 172.21.15.148:59968) 2023-12-21T03:40:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.425+0000 7f30e9bb5700 1 -- 172.21.15.148:0/719424924 learned_addr learned my addr 172.21.15.148:0/719424924 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:02.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.425+0000 7f30e9bb5700 1 -- 172.21.15.148:0/719424924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d4005040 con 0x7f30e40f5410 2023-12-21T03:40:02.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.426+0000 7f30e9bb5700 1 --2- 172.21.15.148:0/719424924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e40f5410 0x7f30e4109430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f30d4013510 tx=0x7f30d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:02.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.426+0000 7f30da7fc700 1 -- 172.21.15.148:0/719424924 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30d4009320 con 0x7f30e40f5410 2023-12-21T03:40:02.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.426+0000 7f30ebe19700 1 -- 172.21.15.148:0/719424924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e4060440 con 0x7f30e40f5410 2023-12-21T03:40:02.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.426+0000 7f30da7fc700 1 -- 172.21.15.148:0/719424924 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30d4013f00 con 0x7f30e40f5410 2023-12-21T03:40:02.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.426+0000 7f30da7fc700 1 -- 172.21.15.148:0/719424924 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30d4014d90 con 0x7f30e40f5410 2023-12-21T03:40:02.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.426+0000 7f30ebe19700 1 -- 172.21.15.148:0/719424924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e4109ae0 con 0x7f30e40f5410 2023-12-21T03:40:02.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.427+0000 7f30da7fc700 1 -- 172.21.15.148:0/719424924 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30d40095f0 con 0x7f30e40f5410 2023-12-21T03:40:02.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.427+0000 7f30ebe19700 1 -- 172.21.15.148:0/719424924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e4045ad0 con 0x7f30e40f5410 2023-12-21T03:40:02.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.427+0000 7f30da7fc700 1 --2- 172.21.15.148:0/719424924 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30d002ff60 0x7f30d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:02.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.427+0000 7f30da7fc700 1 -- 172.21.15.148:0/719424924 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30d40410c0 con 0x7f30e40f5410 2023-12-21T03:40:02.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.427+0000 7f30e93b4700 1 --2- 172.21.15.148:0/719424924 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30d002ff60 0x7f30d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:02.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.427+0000 7f30e93b4700 1 --2- 172.21.15.148:0/719424924 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30d002ff60 0x7f30d0032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f30e00096a0 tx=0x7f30e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:02.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.435+0000 7f30da7fc700 1 -- 172.21.15.148:0/719424924 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30d4014420 con 0x7f30e40f5410 2023-12-21T03:40:02.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.606+0000 7f30ebe19700 1 -- 172.21.15.148:0/719424924 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f30e40f1d70 con 0x7f30d002ff60 2023-12-21T03:40:02.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.608+0000 7f30da7fc700 1 -- 172.21.15.148:0/719424924 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f30e40f1d70 con 0x7f30d002ff60 2023-12-21T03:40:02.609 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:02.609 INFO:teuthology.orchestra.run.smithi148.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "smithi148", "hostname": "smithi148", "labels": [], "status": ""}] 2023-12-21T03:40:02.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.609+0000 7f30ebe19700 1 -- 172.21.15.148:0/719424924 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30d002ff60 msgr2=0x7f30d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:02.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.609+0000 7f30ebe19700 1 --2- 172.21.15.148:0/719424924 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30d002ff60 0x7f30d0032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f30e00096a0 tx=0x7f30e0006b40).stop 2023-12-21T03:40:02.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.609+0000 7f30ebe19700 1 -- 172.21.15.148:0/719424924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e40f5410 msgr2=0x7f30e4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:02.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.609+0000 7f30ebe19700 1 --2- 172.21.15.148:0/719424924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e40f5410 0x7f30e4109430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f30d4013510 tx=0x7f30d400b040).stop 2023-12-21T03:40:02.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.609+0000 7f30ebe19700 1 -- 172.21.15.148:0/719424924 shutdown_connections 2023-12-21T03:40:02.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.609+0000 7f30ebe19700 1 --2- 172.21.15.148:0/719424924 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30d002ff60 0x7f30d0032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:02.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.609+0000 7f30ebe19700 1 --2- 172.21.15.148:0/719424924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e40f5410 0x7f30e4109430 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:02.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.609+0000 7f30ebe19700 1 -- 172.21.15.148:0/719424924 >> 172.21.15.148:0/719424924 conn(0x7f30e40f0b70 msgr2=0x7f30e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:02.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.609+0000 7f30ebe19700 1 -- 172.21.15.148:0/719424924 shutdown_connections 2023-12-21T03:40:02.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:02.609+0000 7f30ebe19700 1 -- 172.21.15.148:0/719424924 wait complete. 2023-12-21T03:40:02.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:02.405054+0000 mgr.smithi078.zidbgq (mgr.14162) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:03.010 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-21T03:40:03.010 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd crush tunables default 2023-12-21T03:40:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:02.608271+0000 mgr.smithi078.zidbgq (mgr.14162) 18 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-21T03:40:05.317 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:04.405311+0000 mgr.smithi078.zidbgq (mgr.14162) 19 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:05.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.890+0000 7f5b07a38700 1 -- 172.21.15.78:0/610498259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b000f6990 msgr2=0x7f5b000f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:05.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.890+0000 7f5b07a38700 1 --2- 172.21.15.78:0/610498259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b000f6990 0x7f5b000f6db0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f5afc009fb0 tx=0x7f5afc009b30).stop 2023-12-21T03:40:05.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.890+0000 7f5b07a38700 1 -- 172.21.15.78:0/610498259 shutdown_connections 2023-12-21T03:40:05.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.890+0000 7f5b07a38700 1 --2- 172.21.15.78:0/610498259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b000f6990 0x7f5b000f6db0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:05.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.890+0000 7f5b07a38700 1 -- 172.21.15.78:0/610498259 >> 172.21.15.78:0/610498259 conn(0x7f5b000f0260 msgr2=0x7f5b000f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:05.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.890+0000 7f5b07a38700 1 -- 172.21.15.78:0/610498259 shutdown_connections 2023-12-21T03:40:05.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.890+0000 7f5b07a38700 1 -- 172.21.15.78:0/610498259 wait complete. 2023-12-21T03:40:05.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.891+0000 7f5b07a38700 1 Processor -- start 2023-12-21T03:40:05.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.891+0000 7f5b07a38700 1 -- start start 2023-12-21T03:40:05.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.891+0000 7f5b07a38700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b000f6990 0x7f5b00106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:05.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.891+0000 7f5b07a38700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b00106c20 con 0x7f5b000f6990 2023-12-21T03:40:05.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.891+0000 7f5b057d4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b000f6990 0x7f5b00106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:05.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.891+0000 7f5b057d4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b000f6990 0x7f5b00106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33926/0 (socket says 172.21.15.78:33926) 2023-12-21T03:40:05.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.891+0000 7f5b057d4700 1 -- 172.21.15.78:0/2075317230 learned_addr learned my addr 172.21.15.78:0/2075317230 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:40:05.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.891+0000 7f5b057d4700 1 -- 172.21.15.78:0/2075317230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc005040 con 0x7f5b000f6990 2023-12-21T03:40:05.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.891+0000 7f5b057d4700 1 --2- 172.21.15.78:0/2075317230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b000f6990 0x7f5b00106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5afc013510 tx=0x7f5afc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:05.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.891+0000 7f5af5ffb700 1 -- 172.21.15.78:0/2075317230 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5afc009320 con 0x7f5b000f6990 2023-12-21T03:40:05.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.892+0000 7f5af5ffb700 1 -- 172.21.15.78:0/2075317230 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5afc013f00 con 0x7f5b000f6990 2023-12-21T03:40:05.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.892+0000 7f5af5ffb700 1 -- 172.21.15.78:0/2075317230 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5afc014d90 con 0x7f5b000f6990 2023-12-21T03:40:05.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.892+0000 7f5b07a38700 1 -- 172.21.15.78:0/2075317230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b00106e20 con 0x7f5b000f6990 2023-12-21T03:40:05.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.892+0000 7f5b07a38700 1 -- 172.21.15.78:0/2075317230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b001072b0 con 0x7f5b000f6990 2023-12-21T03:40:05.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.892+0000 7f5af5ffb700 1 -- 172.21.15.78:0/2075317230 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5afc0263a0 con 0x7f5b000f6990 2023-12-21T03:40:05.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.893+0000 7f5af5ffb700 1 --2- 172.21.15.78:0/2075317230 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5aec02ffb0 0x7f5aec032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:05.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.893+0000 7f5af5ffb700 1 -- 172.21.15.78:0/2075317230 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5afc040f80 con 0x7f5b000f6990 2023-12-21T03:40:05.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.893+0000 7f5b07a38700 1 -- 172.21.15.78:0/2075317230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b00045ad0 con 0x7f5b000f6990 2023-12-21T03:40:05.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.893+0000 7f5b04fd3700 1 --2- 172.21.15.78:0/2075317230 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5aec02ffb0 0x7f5aec032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:05.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.893+0000 7f5b04fd3700 1 --2- 172.21.15.78:0/2075317230 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5aec02ffb0 0x7f5aec032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f5af00096a0 tx=0x7f5af0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:05.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:05.900+0000 7f5af5ffb700 1 -- 172.21.15.78:0/2075317230 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5afc012920 con 0x7f5b000f6990 2023-12-21T03:40:06.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.067+0000 7f5b07a38700 1 -- 172.21.15.78:0/2075317230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5b000faf50 con 0x7f5b000f6990 2023-12-21T03:40:06.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.220+0000 7f5af5ffb700 1 -- 172.21.15.78:0/2075317230 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5afc02b030 con 0x7f5b000f6990 2023-12-21T03:40:06.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.222+0000 7f5b07a38700 1 -- 172.21.15.78:0/2075317230 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5aec02ffb0 msgr2=0x7f5aec032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:06.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.222+0000 7f5b07a38700 1 --2- 172.21.15.78:0/2075317230 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5aec02ffb0 0x7f5aec032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f5af00096a0 tx=0x7f5af0006b40).stop 2023-12-21T03:40:06.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.222+0000 7f5b07a38700 1 -- 172.21.15.78:0/2075317230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b000f6990 msgr2=0x7f5b00106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:06.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.222+0000 7f5b07a38700 1 --2- 172.21.15.78:0/2075317230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b000f6990 0x7f5b00106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5afc013510 tx=0x7f5afc00b040).stop 2023-12-21T03:40:06.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.222+0000 7f5b07a38700 1 -- 172.21.15.78:0/2075317230 shutdown_connections 2023-12-21T03:40:06.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.222+0000 7f5b07a38700 1 --2- 172.21.15.78:0/2075317230 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5aec02ffb0 0x7f5aec032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:06.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.222+0000 7f5b07a38700 1 --2- 172.21.15.78:0/2075317230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b000f6990 0x7f5b00106730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:06.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.222+0000 7f5b07a38700 1 -- 172.21.15.78:0/2075317230 >> 172.21.15.78:0/2075317230 conn(0x7f5b000f0260 msgr2=0x7f5b000f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:06.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.222+0000 7f5b07a38700 1 -- 172.21.15.78:0/2075317230 shutdown_connections 2023-12-21T03:40:06.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:40:06.222+0000 7f5b07a38700 1 -- 172.21.15.78:0/2075317230 wait complete. 2023-12-21T03:40:06.225 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2023-12-21T03:40:06.529 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:06 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:06.069392+0000 mon.smithi078 (mon.0) 126 : audit [INF] from='client.? 172.21.15.78:0/2075317230' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-21T03:40:06.697 INFO:tasks.cephadm:Adding mon.smithi078 on smithi078 2023-12-21T03:40:06.698 INFO:tasks.cephadm:Adding mon.smithi148 on smithi148 2023-12-21T03:40:06.698 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph orch apply mon '2;smithi078:172.21.15.78=smithi078;smithi148:172.21.15.148=smithi148' 2023-12-21T03:40:07.388 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:06.221599+0000 mon.smithi078 (mon.0) 127 : audit [INF] from='client.? 172.21.15.78:0/2075317230' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-21T03:40:07.388 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:06.221646+0000 mon.smithi078 (mon.0) 128 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-12-21T03:40:07.388 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:06.405611+0000 mgr.smithi078.zidbgq (mgr.14162) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:07.388 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:07.213198+0000 mon.smithi078 (mon.0) 129 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:40:07.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.810+0000 7f5171d41700 1 -- 172.21.15.148:0/1675829665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c0f6510 msgr2=0x7f516c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.810+0000 7f5171d41700 1 --2- 172.21.15.148:0/1675829665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c0f6510 0x7f516c0f6930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f5160009fb0 tx=0x7f5160009b30).stop 2023-12-21T03:40:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.810+0000 7f5171d41700 1 -- 172.21.15.148:0/1675829665 shutdown_connections 2023-12-21T03:40:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.810+0000 7f5171d41700 1 --2- 172.21.15.148:0/1675829665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c0f6510 0x7f516c0f6930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.810+0000 7f5171d41700 1 -- 172.21.15.148:0/1675829665 >> 172.21.15.148:0/1675829665 conn(0x7f516c0f1c70 msgr2=0x7f516c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.810+0000 7f5171d41700 1 -- 172.21.15.148:0/1675829665 shutdown_connections 2023-12-21T03:40:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.810+0000 7f5171d41700 1 -- 172.21.15.148:0/1675829665 wait complete. 2023-12-21T03:40:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.810+0000 7f5171d41700 1 Processor -- start 2023-12-21T03:40:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.810+0000 7f5171d41700 1 -- start start 2023-12-21T03:40:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f5171d41700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c0f6510 0x7f516c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f5171d41700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c069e20 con 0x7f516c0f6510 2023-12-21T03:40:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f516b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c0f6510 0x7f516c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f516b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c0f6510 0x7f516c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:59988/0 (socket says 172.21.15.148:59988) 2023-12-21T03:40:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f516b7fe700 1 -- 172.21.15.148:0/832890997 learned_addr learned my addr 172.21.15.148:0/832890997 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f516b7fe700 1 -- 172.21.15.148:0/832890997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5160005040 con 0x7f516c0f6510 2023-12-21T03:40:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f516b7fe700 1 --2- 172.21.15.148:0/832890997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c0f6510 0x7f516c06b9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5160013510 tx=0x7f516000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f5153fff700 1 -- 172.21.15.148:0/832890997 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5160009320 con 0x7f516c0f6510 2023-12-21T03:40:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f5153fff700 1 -- 172.21.15.148:0/832890997 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5160013f00 con 0x7f516c0f6510 2023-12-21T03:40:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f5171d41700 1 -- 172.21.15.148:0/832890997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516c069fc0 con 0x7f516c0f6510 2023-12-21T03:40:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f5153fff700 1 -- 172.21.15.148:0/832890997 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5160014d90 con 0x7f516c0f6510 2023-12-21T03:40:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.811+0000 7f5171d41700 1 -- 172.21.15.148:0/832890997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516c06a450 con 0x7f516c0f6510 2023-12-21T03:40:07.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.812+0000 7f5153fff700 1 -- 172.21.15.148:0/832890997 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51600263a0 con 0x7f516c0f6510 2023-12-21T03:40:07.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.812+0000 7f5153fff700 1 --2- 172.21.15.148:0/832890997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f515402ff60 0x7f5154032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:07.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.812+0000 7f5153fff700 1 -- 172.21.15.148:0/832890997 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5160041300 con 0x7f516c0f6510 2023-12-21T03:40:07.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.812+0000 7f5171d41700 1 -- 172.21.15.148:0/832890997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516c059100 con 0x7f516c0f6510 2023-12-21T03:40:07.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.812+0000 7f516affd700 1 --2- 172.21.15.148:0/832890997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f515402ff60 0x7f5154032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:07.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.813+0000 7f516affd700 1 --2- 172.21.15.148:0/832890997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f515402ff60 0x7f5154032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f515c0096a0 tx=0x7f515c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:07.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.820+0000 7f5153fff700 1 -- 172.21.15.148:0/832890997 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5160012920 con 0x7f516c0f6510 2023-12-21T03:40:07.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:07.998+0000 7f5171d41700 1 -- 172.21.15.148:0/832890997 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi078:172.21.15.78=smithi078;smithi148:172.21.15.148=smithi148", "target": ["mon-mgr", ""]}) v1 -- 0x7f516c06ac70 con 0x7f515402ff60 2023-12-21T03:40:08.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:08.003+0000 7f5153fff700 1 -- 172.21.15.148:0/832890997 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f516c06ac70 con 0x7f515402ff60 2023-12-21T03:40:08.004 INFO:teuthology.orchestra.run.smithi148.stdout:Scheduled mon update... 2023-12-21T03:40:08.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:08.004+0000 7f5171d41700 1 -- 172.21.15.148:0/832890997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f515402ff60 msgr2=0x7f5154032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:08.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:08.004+0000 7f5171d41700 1 --2- 172.21.15.148:0/832890997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f515402ff60 0x7f5154032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f515c0096a0 tx=0x7f515c006b40).stop 2023-12-21T03:40:08.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:08.004+0000 7f5171d41700 1 -- 172.21.15.148:0/832890997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c0f6510 msgr2=0x7f516c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:08.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:08.004+0000 7f5171d41700 1 --2- 172.21.15.148:0/832890997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c0f6510 0x7f516c06b9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5160013510 tx=0x7f516000b040).stop 2023-12-21T03:40:08.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:08.004+0000 7f5171d41700 1 -- 172.21.15.148:0/832890997 shutdown_connections 2023-12-21T03:40:08.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:08.004+0000 7f5171d41700 1 --2- 172.21.15.148:0/832890997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f515402ff60 0x7f5154032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:08.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:08.004+0000 7f5171d41700 1 --2- 172.21.15.148:0/832890997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c0f6510 0x7f516c06b9a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:08.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:08.004+0000 7f5171d41700 1 -- 172.21.15.148:0/832890997 >> 172.21.15.148:0/832890997 conn(0x7f516c0f1c70 msgr2=0x7f516c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:08.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:08.004+0000 7f5171d41700 1 -- 172.21.15.148:0/832890997 shutdown_connections 2023-12-21T03:40:08.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:08.004+0000 7f5171d41700 1 -- 172.21.15.148:0/832890997 wait complete. 2023-12-21T03:40:08.374 DEBUG:teuthology.orchestra.run.smithi148:mon.smithi148> sudo journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi148.service 2023-12-21T03:40:08.377 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:08.377 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:08.404 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:-- Logs begin at Thu 2023-12-21 03:26:40 UTC. -- 2023-12-21T03:40:08.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:40:07.214489+0000 mgr.smithi078.zidbgq (mgr.14162) 21 : cephadm [INF] Deploying daemon node-exporter.smithi078 on smithi078 2023-12-21T03:40:08.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:08.003170+0000 mon.smithi078 (mon.0) 130 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:40:09.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.523+0000 7f1b709d8700 1 -- 172.21.15.148:0/727616265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c0f51a0 msgr2=0x7f1b6c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:09.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.523+0000 7f1b709d8700 1 --2- 172.21.15.148:0/727616265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c0f51a0 0x7f1b6c0f55c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1b5c009fb0 tx=0x7f1b5c009b30).stop 2023-12-21T03:40:09.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.523+0000 7f1b709d8700 1 -- 172.21.15.148:0/727616265 shutdown_connections 2023-12-21T03:40:09.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.523+0000 7f1b709d8700 1 --2- 172.21.15.148:0/727616265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c0f51a0 0x7f1b6c0f55c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:09.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.523+0000 7f1b709d8700 1 -- 172.21.15.148:0/727616265 >> 172.21.15.148:0/727616265 conn(0x7f1b6c0f09a0 msgr2=0x7f1b6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:09.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.523+0000 7f1b709d8700 1 -- 172.21.15.148:0/727616265 shutdown_connections 2023-12-21T03:40:09.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.523+0000 7f1b709d8700 1 -- 172.21.15.148:0/727616265 wait complete. 2023-12-21T03:40:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.524+0000 7f1b709d8700 1 Processor -- start 2023-12-21T03:40:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.524+0000 7f1b709d8700 1 -- start start 2023-12-21T03:40:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.524+0000 7f1b709d8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c0f51a0 0x7f1b6c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.524+0000 7f1b709d8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c060240 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.525+0000 7f1b6ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c0f51a0 0x7f1b6c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.525+0000 7f1b6ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c0f51a0 0x7f1b6c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46516/0 (socket says 172.21.15.148:46516) 2023-12-21T03:40:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.525+0000 7f1b6ad9d700 1 -- 172.21.15.148:0/3423231444 learned_addr learned my addr 172.21.15.148:0/3423231444 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.525+0000 7f1b6ad9d700 1 -- 172.21.15.148:0/3423231444 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c005040 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.525+0000 7f1b6ad9d700 1 --2- 172.21.15.148:0/3423231444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c0f51a0 0x7f1b6c1091b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f1b5c013510 tx=0x7f1b5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.525+0000 7f1b537fe700 1 -- 172.21.15.148:0/3423231444 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b5c009320 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.525+0000 7f1b709d8700 1 -- 172.21.15.148:0/3423231444 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c060440 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.525+0000 7f1b537fe700 1 -- 172.21.15.148:0/3423231444 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b5c013f00 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.525+0000 7f1b709d8700 1 -- 172.21.15.148:0/3423231444 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c109860 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.526+0000 7f1b537fe700 1 -- 172.21.15.148:0/3423231444 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b5c014dc0 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.526+0000 7f1b537fe700 1 -- 172.21.15.148:0/3423231444 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b5c01d3d0 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.526+0000 7f1b537fe700 1 --2- 172.21.15.148:0/3423231444 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1b5402ffb0 0x7f1b54032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:09.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.526+0000 7f1b709d8700 1 -- 172.21.15.148:0/3423231444 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6c045ad0 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.527+0000 7f1b537fe700 1 -- 172.21.15.148:0/3423231444 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b5c0410e0 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.527+0000 7f1b6a59c700 1 --2- 172.21.15.148:0/3423231444 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1b5402ffb0 0x7f1b54032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:09.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.527+0000 7f1b6a59c700 1 --2- 172.21.15.148:0/3423231444 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1b5402ffb0 0x7f1b54032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f1b600096a0 tx=0x7f1b60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:09.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.532+0000 7f1b537fe700 1 -- 172.21.15.148:0/3423231444 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b5c012920 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:07.999805+0000 mgr.smithi078.zidbgq (mgr.14162) 22 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi078:172.21.15.78=smithi078;smithi148:172.21.15.148=smithi148", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:40:09.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:40:08.000895+0000 mgr.smithi078.zidbgq (mgr.14162) 23 : cephadm [INF] Saving service mon spec with placement smithi078:172.21.15.78=smithi078;smithi148:172.21.15.148=smithi148;count:2 2023-12-21T03:40:09.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:08.405872+0000 mgr.smithi078.zidbgq (mgr.14162) 24 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:09.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.736+0000 7f1b709d8700 1 -- 172.21.15.148:0/3423231444 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b6c10a070 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.737+0000 7f1b537fe700 1 -- 172.21.15.148:0/3423231444 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b5c03a030 con 0x7f1b6c0f51a0 2023-12-21T03:40:09.738 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:09.738 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:09.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.738+0000 7f1b709d8700 1 -- 172.21.15.148:0/3423231444 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1b5402ffb0 msgr2=0x7f1b54032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:09.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.738+0000 7f1b709d8700 1 --2- 172.21.15.148:0/3423231444 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1b5402ffb0 0x7f1b54032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f1b600096a0 tx=0x7f1b60006b40).stop 2023-12-21T03:40:09.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.739+0000 7f1b709d8700 1 -- 172.21.15.148:0/3423231444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c0f51a0 msgr2=0x7f1b6c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:09.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.739+0000 7f1b709d8700 1 --2- 172.21.15.148:0/3423231444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c0f51a0 0x7f1b6c1091b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f1b5c013510 tx=0x7f1b5c00b040).stop 2023-12-21T03:40:09.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.739+0000 7f1b709d8700 1 -- 172.21.15.148:0/3423231444 shutdown_connections 2023-12-21T03:40:09.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.739+0000 7f1b709d8700 1 --2- 172.21.15.148:0/3423231444 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1b5402ffb0 0x7f1b54032470 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:09.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.739+0000 7f1b709d8700 1 --2- 172.21.15.148:0/3423231444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c0f51a0 0x7f1b6c1091b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:09.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.739+0000 7f1b709d8700 1 -- 172.21.15.148:0/3423231444 >> 172.21.15.148:0/3423231444 conn(0x7f1b6c0f09a0 msgr2=0x7f1b6c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:09.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.739+0000 7f1b709d8700 1 -- 172.21.15.148:0/3423231444 shutdown_connections 2023-12-21T03:40:09.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:09.739+0000 7f1b709d8700 1 -- 172.21.15.148:0/3423231444 wait complete. 2023-12-21T03:40:09.741 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:10.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:10 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:09.737789+0000 mon.smithi078 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.148:0/3423231444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:11.127 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:11.128 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:11.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:10.406119+0000 mgr.smithi078.zidbgq (mgr.14162) 25 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:12.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.282+0000 7f87877f2700 1 -- 172.21.15.148:0/3651432188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87800601e0 msgr2=0x7f87800f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:12.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.282+0000 7f87877f2700 1 --2- 172.21.15.148:0/3651432188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87800601e0 0x7f87800f9190 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f877c009fb0 tx=0x7f877c009b30).stop 2023-12-21T03:40:12.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.282+0000 7f87877f2700 1 -- 172.21.15.148:0/3651432188 shutdown_connections 2023-12-21T03:40:12.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.282+0000 7f87877f2700 1 --2- 172.21.15.148:0/3651432188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87800601e0 0x7f87800f9190 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:12.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.282+0000 7f87877f2700 1 -- 172.21.15.148:0/3651432188 >> 172.21.15.148:0/3651432188 conn(0x7f87800f0b70 msgr2=0x7f87800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:12.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.282+0000 7f87877f2700 1 -- 172.21.15.148:0/3651432188 shutdown_connections 2023-12-21T03:40:12.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.282+0000 7f87877f2700 1 -- 172.21.15.148:0/3651432188 wait complete. 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.283+0000 7f87877f2700 1 Processor -- start 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.283+0000 7f87877f2700 1 -- start start 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.283+0000 7f87877f2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87800601e0 0x7f87801066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.283+0000 7f87877f2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8780106bd0 con 0x7f87800601e0 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.283+0000 7f878558e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87800601e0 0x7f87801066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.283+0000 7f878558e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87800601e0 0x7f87801066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46534/0 (socket says 172.21.15.148:46534) 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.283+0000 7f878558e700 1 -- 172.21.15.148:0/1235654790 learned_addr learned my addr 172.21.15.148:0/1235654790 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.283+0000 7f878558e700 1 -- 172.21.15.148:0/1235654790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c005040 con 0x7f87800601e0 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.283+0000 7f878558e700 1 --2- 172.21.15.148:0/1235654790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87800601e0 0x7f87801066e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f877c004dd0 tx=0x7f877c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.284+0000 7f8775ffb700 1 -- 172.21.15.148:0/1235654790 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f877c018070 con 0x7f87800601e0 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.284+0000 7f87877f2700 1 -- 172.21.15.148:0/1235654790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8780106dd0 con 0x7f87800601e0 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.284+0000 7f8775ffb700 1 -- 172.21.15.148:0/1235654790 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f877c013410 con 0x7f87800601e0 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.284+0000 7f8775ffb700 1 -- 172.21.15.148:0/1235654790 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f877c01c3c0 con 0x7f87800601e0 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.284+0000 7f87877f2700 1 -- 172.21.15.148:0/1235654790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8780107260 con 0x7f87800601e0 2023-12-21T03:40:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.285+0000 7f8775ffb700 1 -- 172.21.15.148:0/1235654790 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f877c01d420 con 0x7f87800601e0 2023-12-21T03:40:12.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.285+0000 7f87877f2700 1 -- 172.21.15.148:0/1235654790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8780045ad0 con 0x7f87800601e0 2023-12-21T03:40:12.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.285+0000 7f8775ffb700 1 --2- 172.21.15.148:0/1235654790 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f876c02ff60 0x7f876c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:12.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.285+0000 7f8775ffb700 1 -- 172.21.15.148:0/1235654790 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f877c041450 con 0x7f87800601e0 2023-12-21T03:40:12.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.285+0000 7f8784d8d700 1 --2- 172.21.15.148:0/1235654790 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f876c02ff60 0x7f876c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:12.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.285+0000 7f8784d8d700 1 --2- 172.21.15.148:0/1235654790 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f876c02ff60 0x7f876c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f87700096a0 tx=0x7f8770006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:12.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.289+0000 7f8775ffb700 1 -- 172.21.15.148:0/1235654790 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f877c01d820 con 0x7f87800601e0 2023-12-21T03:40:12.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.498+0000 7f87877f2700 1 -- 172.21.15.148:0/1235654790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8780108110 con 0x7f87800601e0 2023-12-21T03:40:12.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.498+0000 7f8775ffb700 1 -- 172.21.15.148:0/1235654790 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f877c01b370 con 0x7f87800601e0 2023-12-21T03:40:12.500 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:12.500 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:12.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.500+0000 7f87877f2700 1 -- 172.21.15.148:0/1235654790 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f876c02ff60 msgr2=0x7f876c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:12.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.500+0000 7f87877f2700 1 --2- 172.21.15.148:0/1235654790 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f876c02ff60 0x7f876c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f87700096a0 tx=0x7f8770006b40).stop 2023-12-21T03:40:12.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.500+0000 7f87877f2700 1 -- 172.21.15.148:0/1235654790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87800601e0 msgr2=0x7f87801066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:12.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.500+0000 7f87877f2700 1 --2- 172.21.15.148:0/1235654790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87800601e0 0x7f87801066e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f877c004dd0 tx=0x7f877c009b30).stop 2023-12-21T03:40:12.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.500+0000 7f87877f2700 1 -- 172.21.15.148:0/1235654790 shutdown_connections 2023-12-21T03:40:12.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.500+0000 7f87877f2700 1 --2- 172.21.15.148:0/1235654790 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f876c02ff60 0x7f876c032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:12.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.500+0000 7f87877f2700 1 --2- 172.21.15.148:0/1235654790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87800601e0 0x7f87801066e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:12.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.500+0000 7f87877f2700 1 -- 172.21.15.148:0/1235654790 >> 172.21.15.148:0/1235654790 conn(0x7f87800f0b70 msgr2=0x7f87800f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:12.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.500+0000 7f87877f2700 1 -- 172.21.15.148:0/1235654790 shutdown_connections 2023-12-21T03:40:12.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:12.500+0000 7f87877f2700 1 -- 172.21.15.148:0/1235654790 wait complete. 2023-12-21T03:40:12.503 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:12.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:12.406510+0000 mgr.smithi078.zidbgq (mgr.14162) 26 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:12.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:12.499304+0000 mon.smithi078 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.148:0/1235654790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:13.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:13.873 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:15.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.075+0000 7f33348c2700 1 -- 172.21.15.148:0/305435715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33300601e0 msgr2=0x7f33300f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:15.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.075+0000 7f33348c2700 1 --2- 172.21.15.148:0/305435715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33300601e0 0x7f33300f8f20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f3320009fb0 tx=0x7f3320009b30).stop 2023-12-21T03:40:15.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.075+0000 7f33348c2700 1 -- 172.21.15.148:0/305435715 shutdown_connections 2023-12-21T03:40:15.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.075+0000 7f33348c2700 1 --2- 172.21.15.148:0/305435715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33300601e0 0x7f33300f8f20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:15.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.075+0000 7f33348c2700 1 -- 172.21.15.148:0/305435715 >> 172.21.15.148:0/305435715 conn(0x7f33300f09a0 msgr2=0x7f33300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:15.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.075+0000 7f33348c2700 1 -- 172.21.15.148:0/305435715 shutdown_connections 2023-12-21T03:40:15.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.075+0000 7f33348c2700 1 -- 172.21.15.148:0/305435715 wait complete. 2023-12-21T03:40:15.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.076+0000 7f33348c2700 1 Processor -- start 2023-12-21T03:40:15.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.076+0000 7f33348c2700 1 -- start start 2023-12-21T03:40:15.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.076+0000 7f33348c2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33300601e0 0x7f3330106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:15.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.076+0000 7f33348c2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33301069f0 con 0x7f33300601e0 2023-12-21T03:40:15.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.076+0000 7f332ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33300601e0 0x7f3330106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:15.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.076+0000 7f332ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33300601e0 0x7f3330106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46554/0 (socket says 172.21.15.148:46554) 2023-12-21T03:40:15.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.076+0000 7f332ed9d700 1 -- 172.21.15.148:0/615022406 learned_addr learned my addr 172.21.15.148:0/615022406 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:15.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.077+0000 7f332ed9d700 1 -- 172.21.15.148:0/615022406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3320005040 con 0x7f33300601e0 2023-12-21T03:40:15.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.077+0000 7f332ed9d700 1 --2- 172.21.15.148:0/615022406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33300601e0 0x7f3330106500 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3320013510 tx=0x7f332000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:15.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.077+0000 7f33177fe700 1 -- 172.21.15.148:0/615022406 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3320009320 con 0x7f33300601e0 2023-12-21T03:40:15.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.077+0000 7f33348c2700 1 -- 172.21.15.148:0/615022406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3330106bf0 con 0x7f33300601e0 2023-12-21T03:40:15.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.077+0000 7f33177fe700 1 -- 172.21.15.148:0/615022406 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3320013f00 con 0x7f33300601e0 2023-12-21T03:40:15.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.077+0000 7f33177fe700 1 -- 172.21.15.148:0/615022406 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3320014d90 con 0x7f33300601e0 2023-12-21T03:40:15.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.077+0000 7f33348c2700 1 -- 172.21.15.148:0/615022406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3330107080 con 0x7f33300601e0 2023-12-21T03:40:15.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.078+0000 7f33177fe700 1 -- 172.21.15.148:0/615022406 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33200263a0 con 0x7f33300601e0 2023-12-21T03:40:15.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.078+0000 7f33348c2700 1 -- 172.21.15.148:0/615022406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3330045ad0 con 0x7f33300601e0 2023-12-21T03:40:15.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.078+0000 7f33177fe700 1 --2- 172.21.15.148:0/615022406 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f331802ff60 0x7f3318032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:15.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.078+0000 7f33177fe700 1 -- 172.21.15.148:0/615022406 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33200412c0 con 0x7f33300601e0 2023-12-21T03:40:15.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.078+0000 7f332e59c700 1 --2- 172.21.15.148:0/615022406 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f331802ff60 0x7f3318032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:15.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.079+0000 7f332e59c700 1 --2- 172.21.15.148:0/615022406 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f331802ff60 0x7f3318032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f33240096a0 tx=0x7f3324006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:15.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.083+0000 7f33177fe700 1 -- 172.21.15.148:0/615022406 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3320012330 con 0x7f33300601e0 2023-12-21T03:40:15.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.287+0000 7f33348c2700 1 -- 172.21.15.148:0/615022406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3330107f30 con 0x7f33300601e0 2023-12-21T03:40:15.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.288+0000 7f33177fe700 1 -- 172.21.15.148:0/615022406 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3320012330 con 0x7f33300601e0 2023-12-21T03:40:15.289 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:15.290 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.290+0000 7f33348c2700 1 -- 172.21.15.148:0/615022406 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f331802ff60 msgr2=0x7f3318032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.290+0000 7f33348c2700 1 --2- 172.21.15.148:0/615022406 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f331802ff60 0x7f3318032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f33240096a0 tx=0x7f3324006b40).stop 2023-12-21T03:40:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.290+0000 7f33348c2700 1 -- 172.21.15.148:0/615022406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33300601e0 msgr2=0x7f3330106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.290+0000 7f33348c2700 1 --2- 172.21.15.148:0/615022406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33300601e0 0x7f3330106500 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3320013510 tx=0x7f332000b040).stop 2023-12-21T03:40:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.290+0000 7f33348c2700 1 -- 172.21.15.148:0/615022406 shutdown_connections 2023-12-21T03:40:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.290+0000 7f33348c2700 1 --2- 172.21.15.148:0/615022406 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f331802ff60 0x7f3318032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.290+0000 7f33348c2700 1 --2- 172.21.15.148:0/615022406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33300601e0 0x7f3330106500 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.290+0000 7f33348c2700 1 -- 172.21.15.148:0/615022406 >> 172.21.15.148:0/615022406 conn(0x7f33300f09a0 msgr2=0x7f33300f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.290+0000 7f33348c2700 1 -- 172.21.15.148:0/615022406 shutdown_connections 2023-12-21T03:40:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:15.290+0000 7f33348c2700 1 -- 172.21.15.148:0/615022406 wait complete. 2023-12-21T03:40:15.292 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:15.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:14.406994+0000 mgr.smithi078.zidbgq (mgr.14162) 27 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:16.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:16 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:15.288996+0000 mon.smithi078 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.148:0/615022406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:16.685 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:16.686 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:17.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:16.407380+0000 mgr.smithi078.zidbgq (mgr.14162) 28 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:17.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.850+0000 7f07c23cd700 1 -- 172.21.15.148:0/831854214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07bc0f5410 msgr2=0x7f07bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:17.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.850+0000 7f07c23cd700 1 --2- 172.21.15.148:0/831854214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07bc0f5410 0x7f07bc0f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f07ac009fb0 tx=0x7f07ac009b30).stop 2023-12-21T03:40:17.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.850+0000 7f07c23cd700 1 -- 172.21.15.148:0/831854214 shutdown_connections 2023-12-21T03:40:17.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.850+0000 7f07c23cd700 1 --2- 172.21.15.148:0/831854214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07bc0f5410 0x7f07bc0f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:17.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.850+0000 7f07c23cd700 1 -- 172.21.15.148:0/831854214 >> 172.21.15.148:0/831854214 conn(0x7f07bc0f0b70 msgr2=0x7f07bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:17.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.850+0000 7f07c23cd700 1 -- 172.21.15.148:0/831854214 shutdown_connections 2023-12-21T03:40:17.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.850+0000 7f07c23cd700 1 -- 172.21.15.148:0/831854214 wait complete. 2023-12-21T03:40:17.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.851+0000 7f07c23cd700 1 Processor -- start 2023-12-21T03:40:17.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.851+0000 7f07c23cd700 1 -- start start 2023-12-21T03:40:17.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.851+0000 7f07c23cd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07bc0f5410 0x7f07bc1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:17.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.851+0000 7f07c23cd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc105ac0 con 0x7f07bc0f5410 2023-12-21T03:40:17.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.852+0000 7f07bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07bc0f5410 0x7f07bc1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.852+0000 7f07bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07bc0f5410 0x7f07bc1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46572/0 (socket says 172.21.15.148:46572) 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.852+0000 7f07bbfff700 1 -- 172.21.15.148:0/2264921569 learned_addr learned my addr 172.21.15.148:0/2264921569 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.852+0000 7f07bbfff700 1 -- 172.21.15.148:0/2264921569 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac005040 con 0x7f07bc0f5410 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.852+0000 7f07bbfff700 1 --2- 172.21.15.148:0/2264921569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07bc0f5410 0x7f07bc1055d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f07ac013510 tx=0x7f07ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.852+0000 7f07b8ff9700 1 -- 172.21.15.148:0/2264921569 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07ac009320 con 0x7f07bc0f5410 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.852+0000 7f07c23cd700 1 -- 172.21.15.148:0/2264921569 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07bc105cc0 con 0x7f07bc0f5410 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.852+0000 7f07b8ff9700 1 -- 172.21.15.148:0/2264921569 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07ac013f00 con 0x7f07bc0f5410 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.852+0000 7f07b8ff9700 1 -- 172.21.15.148:0/2264921569 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07ac014d90 con 0x7f07bc0f5410 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.852+0000 7f07c23cd700 1 -- 172.21.15.148:0/2264921569 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07bc106150 con 0x7f07bc0f5410 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.853+0000 7f07b8ff9700 1 -- 172.21.15.148:0/2264921569 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07ac014420 con 0x7f07bc0f5410 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.853+0000 7f07c23cd700 1 -- 172.21.15.148:0/2264921569 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07bc045ad0 con 0x7f07bc0f5410 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.853+0000 7f07b8ff9700 1 --2- 172.21.15.148:0/2264921569 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f07a402ff60 0x7f07a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.853+0000 7f07b8ff9700 1 -- 172.21.15.148:0/2264921569 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07ac041000 con 0x7f07bc0f5410 2023-12-21T03:40:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.854+0000 7f07bb7fe700 1 --2- 172.21.15.148:0/2264921569 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f07a402ff60 0x7f07a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:17.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.854+0000 7f07bb7fe700 1 --2- 172.21.15.148:0/2264921569 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f07a402ff60 0x7f07a4032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f07b00096a0 tx=0x7f07b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:17.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:17.858+0000 7f07b8ff9700 1 -- 172.21.15.148:0/2264921569 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07ac0095f0 con 0x7f07bc0f5410 2023-12-21T03:40:18.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.067+0000 7f07c23cd700 1 -- 172.21.15.148:0/2264921569 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07bc107000 con 0x7f07bc0f5410 2023-12-21T03:40:18.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.067+0000 7f07b8ff9700 1 -- 172.21.15.148:0/2264921569 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07ac01db70 con 0x7f07bc0f5410 2023-12-21T03:40:18.070 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:18.070 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:18.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.069+0000 7f07c23cd700 1 -- 172.21.15.148:0/2264921569 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f07a402ff60 msgr2=0x7f07a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.069+0000 7f07c23cd700 1 --2- 172.21.15.148:0/2264921569 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f07a402ff60 0x7f07a4032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f07b00096a0 tx=0x7f07b0006b40).stop 2023-12-21T03:40:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.069+0000 7f07c23cd700 1 -- 172.21.15.148:0/2264921569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07bc0f5410 msgr2=0x7f07bc1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.069+0000 7f07c23cd700 1 --2- 172.21.15.148:0/2264921569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07bc0f5410 0x7f07bc1055d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f07ac013510 tx=0x7f07ac00b040).stop 2023-12-21T03:40:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.069+0000 7f07c23cd700 1 -- 172.21.15.148:0/2264921569 shutdown_connections 2023-12-21T03:40:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.069+0000 7f07c23cd700 1 --2- 172.21.15.148:0/2264921569 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f07a402ff60 0x7f07a4032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.069+0000 7f07c23cd700 1 --2- 172.21.15.148:0/2264921569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07bc0f5410 0x7f07bc1055d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.069+0000 7f07c23cd700 1 -- 172.21.15.148:0/2264921569 >> 172.21.15.148:0/2264921569 conn(0x7f07bc0f0b70 msgr2=0x7f07bc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.069+0000 7f07c23cd700 1 -- 172.21.15.148:0/2264921569 shutdown_connections 2023-12-21T03:40:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:18.069+0000 7f07c23cd700 1 -- 172.21.15.148:0/2264921569 wait complete. 2023-12-21T03:40:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:18.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:18 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:18.069212+0000 mon.smithi078 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.148:0/2264921569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:19.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:19.457 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:19.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:18.407630+0000 mgr.smithi078.zidbgq (mgr.14162) 29 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:20.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.581+0000 7f28db0cb700 1 -- 172.21.15.148:0/4062354022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28d4060280 msgr2=0x7f28d40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:20.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.581+0000 7f28db0cb700 1 --2- 172.21.15.148:0/4062354022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28d4060280 0x7f28d40f90f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f28c8009fb0 tx=0x7f28c8009b30).stop 2023-12-21T03:40:20.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.581+0000 7f28db0cb700 1 -- 172.21.15.148:0/4062354022 shutdown_connections 2023-12-21T03:40:20.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.581+0000 7f28db0cb700 1 --2- 172.21.15.148:0/4062354022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28d4060280 0x7f28d40f90f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:20.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.581+0000 7f28db0cb700 1 -- 172.21.15.148:0/4062354022 >> 172.21.15.148:0/4062354022 conn(0x7f28d40f0b70 msgr2=0x7f28d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:20.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.581+0000 7f28db0cb700 1 -- 172.21.15.148:0/4062354022 shutdown_connections 2023-12-21T03:40:20.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.581+0000 7f28db0cb700 1 -- 172.21.15.148:0/4062354022 wait complete. 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.581+0000 7f28db0cb700 1 Processor -- start 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.581+0000 7f28db0cb700 1 -- start start 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28db0cb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28d4060280 0x7f28d40f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28db0cb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40f4ff0 con 0x7f28d4060280 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28d8e67700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28d4060280 0x7f28d40f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28d8e67700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28d4060280 0x7f28d40f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:40202/0 (socket says 172.21.15.148:40202) 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28d8e67700 1 -- 172.21.15.148:0/852085298 learned_addr learned my addr 172.21.15.148:0/852085298 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28d8e67700 1 -- 172.21.15.148:0/852085298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c8005040 con 0x7f28d4060280 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28d8e67700 1 --2- 172.21.15.148:0/852085298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28d4060280 0x7f28d40f6b70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f28c8013510 tx=0x7f28c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28d17fa700 1 -- 172.21.15.148:0/852085298 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28c8009320 con 0x7f28d4060280 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28d17fa700 1 -- 172.21.15.148:0/852085298 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28c8013f00 con 0x7f28d4060280 2023-12-21T03:40:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28db0cb700 1 -- 172.21.15.148:0/852085298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d40f5190 con 0x7f28d4060280 2023-12-21T03:40:20.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28d17fa700 1 -- 172.21.15.148:0/852085298 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28c8014dc0 con 0x7f28d4060280 2023-12-21T03:40:20.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.582+0000 7f28db0cb700 1 -- 172.21.15.148:0/852085298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d40f5620 con 0x7f28d4060280 2023-12-21T03:40:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.583+0000 7f28d17fa700 1 -- 172.21.15.148:0/852085298 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28c80095f0 con 0x7f28d4060280 2023-12-21T03:40:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.583+0000 7f28db0cb700 1 -- 172.21.15.148:0/852085298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d4059100 con 0x7f28d4060280 2023-12-21T03:40:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.583+0000 7f28d17fa700 1 --2- 172.21.15.148:0/852085298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f28c402ff60 0x7f28c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.583+0000 7f28d17fa700 1 -- 172.21.15.148:0/852085298 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f28c80410c0 con 0x7f28d4060280 2023-12-21T03:40:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.584+0000 7f28d3fff700 1 --2- 172.21.15.148:0/852085298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f28c402ff60 0x7f28c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.584+0000 7f28d3fff700 1 --2- 172.21.15.148:0/852085298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f28c402ff60 0x7f28c4032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f28c00096a0 tx=0x7f28c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:20.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.588+0000 7f28d17fa700 1 -- 172.21.15.148:0/852085298 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28c8014420 con 0x7f28d4060280 2023-12-21T03:40:20.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.796+0000 7f28db0cb700 1 -- 172.21.15.148:0/852085298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28d40fbf50 con 0x7f28d4060280 2023-12-21T03:40:20.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.797+0000 7f28d17fa700 1 -- 172.21.15.148:0/852085298 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f28c8010230 con 0x7f28d4060280 2023-12-21T03:40:20.799 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:20.799 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:20.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.798+0000 7f28db0cb700 1 -- 172.21.15.148:0/852085298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f28c402ff60 msgr2=0x7f28c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:20.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.798+0000 7f28db0cb700 1 --2- 172.21.15.148:0/852085298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f28c402ff60 0x7f28c4032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f28c00096a0 tx=0x7f28c0006b40).stop 2023-12-21T03:40:20.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.798+0000 7f28db0cb700 1 -- 172.21.15.148:0/852085298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28d4060280 msgr2=0x7f28d40f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:20.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.798+0000 7f28db0cb700 1 --2- 172.21.15.148:0/852085298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28d4060280 0x7f28d40f6b70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f28c8013510 tx=0x7f28c800b040).stop 2023-12-21T03:40:20.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.799+0000 7f28db0cb700 1 -- 172.21.15.148:0/852085298 shutdown_connections 2023-12-21T03:40:20.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.799+0000 7f28db0cb700 1 --2- 172.21.15.148:0/852085298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f28c402ff60 0x7f28c4032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:20.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.799+0000 7f28db0cb700 1 --2- 172.21.15.148:0/852085298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28d4060280 0x7f28d40f6b70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:20.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.799+0000 7f28db0cb700 1 -- 172.21.15.148:0/852085298 >> 172.21.15.148:0/852085298 conn(0x7f28d40f0b70 msgr2=0x7f28d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:20.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.799+0000 7f28db0cb700 1 -- 172.21.15.148:0/852085298 shutdown_connections 2023-12-21T03:40:20.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:20.799+0000 7f28db0cb700 1 -- 172.21.15.148:0/852085298 wait complete. 2023-12-21T03:40:20.802 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:20.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:20 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:20.408003+0000 mgr.smithi078.zidbgq (mgr.14162) 30 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:20.798818+0000 mon.smithi078 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.148:0/852085298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:22.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:22.213 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:22.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:22.408427+0000 mgr.smithi078.zidbgq (mgr.14162) 31 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:23.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.341+0000 7fc05c036700 1 -- 172.21.15.148:0/1092700083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0540f5410 msgr2=0x7fc0540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:23.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.341+0000 7fc05c036700 1 --2- 172.21.15.148:0/1092700083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0540f5410 0x7fc0540f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc044009fb0 tx=0x7fc044009b30).stop 2023-12-21T03:40:23.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.341+0000 7fc05c036700 1 -- 172.21.15.148:0/1092700083 shutdown_connections 2023-12-21T03:40:23.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.341+0000 7fc05c036700 1 --2- 172.21.15.148:0/1092700083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0540f5410 0x7fc0540f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:23.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.341+0000 7fc05c036700 1 -- 172.21.15.148:0/1092700083 >> 172.21.15.148:0/1092700083 conn(0x7fc0540f0b70 msgr2=0x7fc0540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:23.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.341+0000 7fc05c036700 1 -- 172.21.15.148:0/1092700083 shutdown_connections 2023-12-21T03:40:23.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.341+0000 7fc05c036700 1 -- 172.21.15.148:0/1092700083 wait complete. 2023-12-21T03:40:23.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.341+0000 7fc05c036700 1 Processor -- start 2023-12-21T03:40:23.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.341+0000 7fc05c036700 1 -- start start 2023-12-21T03:40:23.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc05c036700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0540f5410 0x7fc0541093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:23.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc05c036700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc054060280 con 0x7fc0540f5410 2023-12-21T03:40:23.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc059dd2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0540f5410 0x7fc0541093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:23.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc059dd2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0540f5410 0x7fc0541093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:40214/0 (socket says 172.21.15.148:40214) 2023-12-21T03:40:23.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc059dd2700 1 -- 172.21.15.148:0/4222159031 learned_addr learned my addr 172.21.15.148:0/4222159031 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:23.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc059dd2700 1 -- 172.21.15.148:0/4222159031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc044005040 con 0x7fc0540f5410 2023-12-21T03:40:23.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc059dd2700 1 --2- 172.21.15.148:0/4222159031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0540f5410 0x7fc0541093c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc044013510 tx=0x7fc04400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:23.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc04a7fc700 1 -- 172.21.15.148:0/4222159031 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc044009320 con 0x7fc0540f5410 2023-12-21T03:40:23.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc05c036700 1 -- 172.21.15.148:0/4222159031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc054060480 con 0x7fc0540f5410 2023-12-21T03:40:23.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc04a7fc700 1 -- 172.21.15.148:0/4222159031 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc044013f00 con 0x7fc0540f5410 2023-12-21T03:40:23.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc04a7fc700 1 -- 172.21.15.148:0/4222159031 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc044014d90 con 0x7fc0540f5410 2023-12-21T03:40:23.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.342+0000 7fc05c036700 1 -- 172.21.15.148:0/4222159031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc054109af0 con 0x7fc0540f5410 2023-12-21T03:40:23.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.343+0000 7fc04a7fc700 1 -- 172.21.15.148:0/4222159031 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0440263a0 con 0x7fc0540f5410 2023-12-21T03:40:23.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.343+0000 7fc04a7fc700 1 --2- 172.21.15.148:0/4222159031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc04002ff60 0x7fc040032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:23.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.343+0000 7fc04a7fc700 1 -- 172.21.15.148:0/4222159031 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc044041300 con 0x7fc0540f5410 2023-12-21T03:40:23.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.343+0000 7fc05c036700 1 -- 172.21.15.148:0/4222159031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc054045ad0 con 0x7fc0540f5410 2023-12-21T03:40:23.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.344+0000 7fc0595d1700 1 --2- 172.21.15.148:0/4222159031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc04002ff60 0x7fc040032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:23.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.344+0000 7fc0595d1700 1 --2- 172.21.15.148:0/4222159031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc04002ff60 0x7fc040032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc0500096a0 tx=0x7fc050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:23.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.349+0000 7fc04a7fc700 1 -- 172.21.15.148:0/4222159031 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc044012920 con 0x7fc0540f5410 2023-12-21T03:40:23.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.558+0000 7fc05c036700 1 -- 172.21.15.148:0/4222159031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc05410a1a0 con 0x7fc0540f5410 2023-12-21T03:40:23.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.559+0000 7fc04a7fc700 1 -- 172.21.15.148:0/4222159031 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc04402b030 con 0x7fc0540f5410 2023-12-21T03:40:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:23.561 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:23.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.560+0000 7fc05c036700 1 -- 172.21.15.148:0/4222159031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc04002ff60 msgr2=0x7fc040032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:23.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.560+0000 7fc05c036700 1 --2- 172.21.15.148:0/4222159031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc04002ff60 0x7fc040032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc0500096a0 tx=0x7fc050006b40).stop 2023-12-21T03:40:23.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.560+0000 7fc05c036700 1 -- 172.21.15.148:0/4222159031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0540f5410 msgr2=0x7fc0541093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:23.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.560+0000 7fc05c036700 1 --2- 172.21.15.148:0/4222159031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0540f5410 0x7fc0541093c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc044013510 tx=0x7fc04400b040).stop 2023-12-21T03:40:23.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.560+0000 7fc05c036700 1 -- 172.21.15.148:0/4222159031 shutdown_connections 2023-12-21T03:40:23.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.560+0000 7fc05c036700 1 --2- 172.21.15.148:0/4222159031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc04002ff60 0x7fc040032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:23.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.560+0000 7fc05c036700 1 --2- 172.21.15.148:0/4222159031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0540f5410 0x7fc0541093c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:23.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.560+0000 7fc05c036700 1 -- 172.21.15.148:0/4222159031 >> 172.21.15.148:0/4222159031 conn(0x7fc0540f0b70 msgr2=0x7fc0540f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:23.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.560+0000 7fc05c036700 1 -- 172.21.15.148:0/4222159031 shutdown_connections 2023-12-21T03:40:23.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:23.560+0000 7fc05c036700 1 -- 172.21.15.148:0/4222159031 wait complete. 2023-12-21T03:40:23.563 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:24.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:23.560339+0000 mon.smithi078 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.148:0/4222159031' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:24.964 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:24.964 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:25.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:24.408861+0000 mgr.smithi078.zidbgq (mgr.14162) 32 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:26.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.103+0000 7f10a826d700 1 -- 172.21.15.148:0/2152717341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10a00f5410 msgr2=0x7f10a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:26.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.103+0000 7f10a826d700 1 --2- 172.21.15.148:0/2152717341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10a00f5410 0x7f10a00f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f1090009fb0 tx=0x7f1090009b30).stop 2023-12-21T03:40:26.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.104+0000 7f10a826d700 1 -- 172.21.15.148:0/2152717341 shutdown_connections 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.104+0000 7f10a826d700 1 --2- 172.21.15.148:0/2152717341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10a00f5410 0x7f10a00f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.104+0000 7f10a826d700 1 -- 172.21.15.148:0/2152717341 >> 172.21.15.148:0/2152717341 conn(0x7f10a00f0b70 msgr2=0x7f10a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.104+0000 7f10a826d700 1 -- 172.21.15.148:0/2152717341 shutdown_connections 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.104+0000 7f10a826d700 1 -- 172.21.15.148:0/2152717341 wait complete. 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.104+0000 7f10a826d700 1 Processor -- start 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.104+0000 7f10a826d700 1 -- start start 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.104+0000 7f10a826d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10a00f5410 0x7f10a0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.104+0000 7f10a826d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a0060220 con 0x7f10a00f5410 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.104+0000 7f10a6009700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10a00f5410 0x7f10a0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.105+0000 7f10a6009700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10a00f5410 0x7f10a0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:40236/0 (socket says 172.21.15.148:40236) 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.105+0000 7f10a6009700 1 -- 172.21.15.148:0/3189200029 learned_addr learned my addr 172.21.15.148:0/3189200029 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.105+0000 7f10a6009700 1 -- 172.21.15.148:0/3189200029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1090005040 con 0x7f10a00f5410 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.105+0000 7f10a6009700 1 --2- 172.21.15.148:0/3189200029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10a00f5410 0x7f10a0109450 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1090013510 tx=0x7f109000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:26.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.105+0000 7f1096ffd700 1 -- 172.21.15.148:0/3189200029 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1090009320 con 0x7f10a00f5410 2023-12-21T03:40:26.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.105+0000 7f1096ffd700 1 -- 172.21.15.148:0/3189200029 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1090013f00 con 0x7f10a00f5410 2023-12-21T03:40:26.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.105+0000 7f1096ffd700 1 -- 172.21.15.148:0/3189200029 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1090014d90 con 0x7f10a00f5410 2023-12-21T03:40:26.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.105+0000 7f10a826d700 1 -- 172.21.15.148:0/3189200029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a0060420 con 0x7f10a00f5410 2023-12-21T03:40:26.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.105+0000 7f10a826d700 1 -- 172.21.15.148:0/3189200029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a0109b00 con 0x7f10a00f5410 2023-12-21T03:40:26.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.106+0000 7f1096ffd700 1 -- 172.21.15.148:0/3189200029 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10900095f0 con 0x7f10a00f5410 2023-12-21T03:40:26.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.106+0000 7f10a826d700 1 -- 172.21.15.148:0/3189200029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a0059100 con 0x7f10a00f5410 2023-12-21T03:40:26.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.106+0000 7f1096ffd700 1 --2- 172.21.15.148:0/3189200029 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f108c02ff60 0x7f108c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:26.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.106+0000 7f1096ffd700 1 -- 172.21.15.148:0/3189200029 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1090041020 con 0x7f10a00f5410 2023-12-21T03:40:26.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.106+0000 7f10a5808700 1 --2- 172.21.15.148:0/3189200029 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f108c02ff60 0x7f108c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:26.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.107+0000 7f10a5808700 1 --2- 172.21.15.148:0/3189200029 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f108c02ff60 0x7f108c032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f109c0096a0 tx=0x7f109c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:26.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.111+0000 7f1096ffd700 1 -- 172.21.15.148:0/3189200029 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1090014420 con 0x7f10a00f5410 2023-12-21T03:40:26.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.320+0000 7f10a826d700 1 -- 172.21.15.148:0/3189200029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10a0045ad0 con 0x7f10a00f5410 2023-12-21T03:40:26.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.320+0000 7f1096ffd700 1 -- 172.21.15.148:0/3189200029 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f109001d7f0 con 0x7f10a00f5410 2023-12-21T03:40:26.323 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:26.323 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:26.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.322+0000 7f10a826d700 1 -- 172.21.15.148:0/3189200029 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f108c02ff60 msgr2=0x7f108c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:26.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.322+0000 7f10a826d700 1 --2- 172.21.15.148:0/3189200029 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f108c02ff60 0x7f108c032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f109c0096a0 tx=0x7f109c006b40).stop 2023-12-21T03:40:26.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.322+0000 7f10a826d700 1 -- 172.21.15.148:0/3189200029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10a00f5410 msgr2=0x7f10a0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:26.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.322+0000 7f10a826d700 1 --2- 172.21.15.148:0/3189200029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10a00f5410 0x7f10a0109450 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1090013510 tx=0x7f109000b040).stop 2023-12-21T03:40:26.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.322+0000 7f10a826d700 1 -- 172.21.15.148:0/3189200029 shutdown_connections 2023-12-21T03:40:26.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.322+0000 7f10a826d700 1 --2- 172.21.15.148:0/3189200029 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f108c02ff60 0x7f108c032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:26.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.322+0000 7f10a826d700 1 --2- 172.21.15.148:0/3189200029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10a00f5410 0x7f10a0109450 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:26.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.322+0000 7f10a826d700 1 -- 172.21.15.148:0/3189200029 >> 172.21.15.148:0/3189200029 conn(0x7f10a00f0b70 msgr2=0x7f10a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:26.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.322+0000 7f10a826d700 1 -- 172.21.15.148:0/3189200029 shutdown_connections 2023-12-21T03:40:26.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:26.322+0000 7f10a826d700 1 -- 172.21.15.148:0/3189200029 wait complete. 2023-12-21T03:40:26.326 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:26.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:26.322289+0000 mon.smithi078 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.148:0/3189200029' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:27.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:27.712 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:26.409132+0000 mgr.smithi078.zidbgq (mgr.14162) 33 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:28.409431+0000 mgr.smithi078.zidbgq (mgr.14162) 34 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:30.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:30.409713+0000 mgr.smithi078.zidbgq (mgr.14162) 35 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:31.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.956+0000 7f5ccb177700 1 -- 172.21.15.148:0/4182363835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc40f4b80 msgr2=0x7f5cc40f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:31.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.956+0000 7f5ccb177700 1 --2- 172.21.15.148:0/4182363835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc40f4b80 0x7f5cc40f4fa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f5cb8009fb0 tx=0x7f5cb8009b30).stop 2023-12-21T03:40:31.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.956+0000 7f5ccb177700 1 -- 172.21.15.148:0/4182363835 shutdown_connections 2023-12-21T03:40:31.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.956+0000 7f5ccb177700 1 --2- 172.21.15.148:0/4182363835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc40f4b80 0x7f5cc40f4fa0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:31.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.956+0000 7f5ccb177700 1 -- 172.21.15.148:0/4182363835 >> 172.21.15.148:0/4182363835 conn(0x7f5cc40f02e0 msgr2=0x7f5cc40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:31.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.956+0000 7f5ccb177700 1 -- 172.21.15.148:0/4182363835 shutdown_connections 2023-12-21T03:40:31.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.956+0000 7f5ccb177700 1 -- 172.21.15.148:0/4182363835 wait complete. 2023-12-21T03:40:31.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.956+0000 7f5ccb177700 1 Processor -- start 2023-12-21T03:40:31.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.957+0000 7f5ccb177700 1 -- start start 2023-12-21T03:40:31.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.957+0000 7f5ccb177700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc40f4b80 0x7f5cc40fcb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:31.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.957+0000 7f5ccb177700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc40faf90 con 0x7f5cc40f4b80 2023-12-21T03:40:31.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.957+0000 7f5cc8f13700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc40f4b80 0x7f5cc40fcb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.957+0000 7f5cc8f13700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc40f4b80 0x7f5cc40fcb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35394/0 (socket says 172.21.15.148:35394) 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.957+0000 7f5cc8f13700 1 -- 172.21.15.148:0/3801843298 learned_addr learned my addr 172.21.15.148:0/3801843298 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.957+0000 7f5cc8f13700 1 -- 172.21.15.148:0/3801843298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb8005040 con 0x7f5cc40f4b80 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.957+0000 7f5cc8f13700 1 --2- 172.21.15.148:0/3801843298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc40f4b80 0x7f5cc40fcb10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f5cb8014000 tx=0x7f5cb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.958+0000 7f5cc17fa700 1 -- 172.21.15.148:0/3801843298 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cb80095f0 con 0x7f5cc40f4b80 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.958+0000 7f5cc17fa700 1 -- 172.21.15.148:0/3801843298 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cb8007e70 con 0x7f5cc40f4b80 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.958+0000 7f5ccb177700 1 -- 172.21.15.148:0/3801843298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc40fb130 con 0x7f5cc40f4b80 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.958+0000 7f5cc17fa700 1 -- 172.21.15.148:0/3801843298 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cb8014ea0 con 0x7f5cc40f4b80 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.958+0000 7f5ccb177700 1 -- 172.21.15.148:0/3801843298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc40fb5c0 con 0x7f5cc40f4b80 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.958+0000 7f5cc17fa700 1 -- 172.21.15.148:0/3801843298 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cb8025070 con 0x7f5cc40f4b80 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.959+0000 7f5ccb177700 1 -- 172.21.15.148:0/3801843298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc4059100 con 0x7f5cc40f4b80 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.959+0000 7f5cc17fa700 1 --2- 172.21.15.148:0/3801843298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5cb402ff60 0x7f5cb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.959+0000 7f5cc17fa700 1 -- 172.21.15.148:0/3801843298 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cb8040a30 con 0x7f5cc40f4b80 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.959+0000 7f5cc3fff700 1 --2- 172.21.15.148:0/3801843298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5cb402ff60 0x7f5cb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:31.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.959+0000 7f5cc3fff700 1 --2- 172.21.15.148:0/3801843298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5cb402ff60 0x7f5cb4032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f5cb00096a0 tx=0x7f5cb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:31.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:30.964+0000 7f5cc17fa700 1 -- 172.21.15.148:0/3801843298 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cb80079c0 con 0x7f5cc40f4b80 2023-12-21T03:40:31.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.170+0000 7f5ccb177700 1 -- 172.21.15.148:0/3801843298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cc4045ad0 con 0x7f5cc40f4b80 2023-12-21T03:40:31.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.170+0000 7f5cc17fa700 1 -- 172.21.15.148:0/3801843298 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cb800f300 con 0x7f5cc40f4b80 2023-12-21T03:40:31.171 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:31.171 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:31.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.172+0000 7f5ccb177700 1 -- 172.21.15.148:0/3801843298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5cb402ff60 msgr2=0x7f5cb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:31.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.172+0000 7f5ccb177700 1 --2- 172.21.15.148:0/3801843298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5cb402ff60 0x7f5cb4032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f5cb00096a0 tx=0x7f5cb0006b40).stop 2023-12-21T03:40:31.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.172+0000 7f5ccb177700 1 -- 172.21.15.148:0/3801843298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc40f4b80 msgr2=0x7f5cc40fcb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:31.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.172+0000 7f5ccb177700 1 --2- 172.21.15.148:0/3801843298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc40f4b80 0x7f5cc40fcb10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f5cb8014000 tx=0x7f5cb800b040).stop 2023-12-21T03:40:31.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.172+0000 7f5ccb177700 1 -- 172.21.15.148:0/3801843298 shutdown_connections 2023-12-21T03:40:31.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.172+0000 7f5ccb177700 1 --2- 172.21.15.148:0/3801843298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5cb402ff60 0x7f5cb4032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:31.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.172+0000 7f5ccb177700 1 --2- 172.21.15.148:0/3801843298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc40f4b80 0x7f5cc40fcb10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:31.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.172+0000 7f5ccb177700 1 -- 172.21.15.148:0/3801843298 >> 172.21.15.148:0/3801843298 conn(0x7f5cc40f02e0 msgr2=0x7f5cc40f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:31.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.172+0000 7f5ccb177700 1 -- 172.21.15.148:0/3801843298 shutdown_connections 2023-12-21T03:40:31.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:31.172+0000 7f5ccb177700 1 -- 172.21.15.148:0/3801843298 wait complete. 2023-12-21T03:40:31.174 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:30.540330+0000 mon.smithi078 (mon.0) 138 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:40:31.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:30.548199+0000 mon.smithi078 (mon.0) 139 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:40:31.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:31.171088+0000 mon.smithi078 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.148:0/3801843298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:32.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:32.763 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:32.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:32.410082+0000 mgr.smithi078.zidbgq (mgr.14162) 36 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:33.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.920+0000 7ff32b651700 1 -- 172.21.15.148:0/441769519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324060280 msgr2=0x7ff3240f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:33.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.920+0000 7ff32b651700 1 --2- 172.21.15.148:0/441769519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324060280 0x7ff3240f90f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7ff320009fb0 tx=0x7ff320009b30).stop 2023-12-21T03:40:33.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.920+0000 7ff32b651700 1 -- 172.21.15.148:0/441769519 shutdown_connections 2023-12-21T03:40:33.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.920+0000 7ff32b651700 1 --2- 172.21.15.148:0/441769519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324060280 0x7ff3240f90f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.920+0000 7ff32b651700 1 -- 172.21.15.148:0/441769519 >> 172.21.15.148:0/441769519 conn(0x7ff3240f0b70 msgr2=0x7ff3240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.920+0000 7ff32b651700 1 -- 172.21.15.148:0/441769519 shutdown_connections 2023-12-21T03:40:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.920+0000 7ff32b651700 1 -- 172.21.15.148:0/441769519 wait complete. 2023-12-21T03:40:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.921+0000 7ff32b651700 1 Processor -- start 2023-12-21T03:40:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.921+0000 7ff32b651700 1 -- start start 2023-12-21T03:40:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.921+0000 7ff32b651700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324060280 0x7ff3241066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.921+0000 7ff32b651700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff324106bd0 con 0x7ff324060280 2023-12-21T03:40:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.921+0000 7ff3293ed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324060280 0x7ff3241066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.921+0000 7ff3293ed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324060280 0x7ff3241066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35418/0 (socket says 172.21.15.148:35418) 2023-12-21T03:40:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.921+0000 7ff3293ed700 1 -- 172.21.15.148:0/2590526534 learned_addr learned my addr 172.21.15.148:0/2590526534 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:33.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.921+0000 7ff3293ed700 1 -- 172.21.15.148:0/2590526534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff320005040 con 0x7ff324060280 2023-12-21T03:40:33.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.922+0000 7ff3293ed700 1 --2- 172.21.15.148:0/2590526534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324060280 0x7ff3241066e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff320013510 tx=0x7ff32000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:33.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.922+0000 7ff319ffb700 1 -- 172.21.15.148:0/2590526534 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff320009320 con 0x7ff324060280 2023-12-21T03:40:33.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.922+0000 7ff32b651700 1 -- 172.21.15.148:0/2590526534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff324106dd0 con 0x7ff324060280 2023-12-21T03:40:33.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.922+0000 7ff319ffb700 1 -- 172.21.15.148:0/2590526534 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff320013f00 con 0x7ff324060280 2023-12-21T03:40:33.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.922+0000 7ff319ffb700 1 -- 172.21.15.148:0/2590526534 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff320014dc0 con 0x7ff324060280 2023-12-21T03:40:33.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.922+0000 7ff32b651700 1 -- 172.21.15.148:0/2590526534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff324107260 con 0x7ff324060280 2023-12-21T03:40:33.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.923+0000 7ff32b651700 1 -- 172.21.15.148:0/2590526534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff324045ad0 con 0x7ff324060280 2023-12-21T03:40:33.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.923+0000 7ff319ffb700 1 -- 172.21.15.148:0/2590526534 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3200095f0 con 0x7ff324060280 2023-12-21T03:40:33.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.923+0000 7ff319ffb700 1 --2- 172.21.15.148:0/2590526534 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff31002ffb0 0x7ff310032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:33.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.923+0000 7ff319ffb700 1 -- 172.21.15.148:0/2590526534 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff320040fb0 con 0x7ff324060280 2023-12-21T03:40:33.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.923+0000 7ff328bec700 1 --2- 172.21.15.148:0/2590526534 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff31002ffb0 0x7ff310032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:33.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.924+0000 7ff328bec700 1 --2- 172.21.15.148:0/2590526534 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff31002ffb0 0x7ff310032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff3140096a0 tx=0x7ff314006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:33.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:33.928+0000 7ff319ffb700 1 -- 172.21.15.148:0/2590526534 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff32006d2b0 con 0x7ff324060280 2023-12-21T03:40:34.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.135+0000 7ff32b651700 1 -- 172.21.15.148:0/2590526534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff324108110 con 0x7ff324060280 2023-12-21T03:40:34.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.136+0000 7ff319ffb700 1 -- 172.21.15.148:0/2590526534 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff32001d7f0 con 0x7ff324060280 2023-12-21T03:40:34.137 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:34.138 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:34.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.138+0000 7ff32b651700 1 -- 172.21.15.148:0/2590526534 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff31002ffb0 msgr2=0x7ff310032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:34.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.138+0000 7ff32b651700 1 --2- 172.21.15.148:0/2590526534 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff31002ffb0 0x7ff310032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff3140096a0 tx=0x7ff314006b40).stop 2023-12-21T03:40:34.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.138+0000 7ff32b651700 1 -- 172.21.15.148:0/2590526534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324060280 msgr2=0x7ff3241066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:34.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.138+0000 7ff32b651700 1 --2- 172.21.15.148:0/2590526534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324060280 0x7ff3241066e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff320013510 tx=0x7ff32000b040).stop 2023-12-21T03:40:34.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.138+0000 7ff32b651700 1 -- 172.21.15.148:0/2590526534 shutdown_connections 2023-12-21T03:40:34.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.138+0000 7ff32b651700 1 --2- 172.21.15.148:0/2590526534 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff31002ffb0 0x7ff310032470 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.138+0000 7ff32b651700 1 --2- 172.21.15.148:0/2590526534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324060280 0x7ff3241066e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.138+0000 7ff32b651700 1 -- 172.21.15.148:0/2590526534 >> 172.21.15.148:0/2590526534 conn(0x7ff3240f0b70 msgr2=0x7ff3240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.138+0000 7ff32b651700 1 -- 172.21.15.148:0/2590526534 shutdown_connections 2023-12-21T03:40:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:34.138+0000 7ff32b651700 1 -- 172.21.15.148:0/2590526534 wait complete. 2023-12-21T03:40:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:34.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:34 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:34.136902+0000 mon.smithi078 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.148:0/2590526534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:35.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:35.523 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:35.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:34.410487+0000 mgr.smithi078.zidbgq (mgr.14162) 37 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:36.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.654+0000 7f00a79a6700 1 -- 172.21.15.148:0/2395830180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a00f6510 msgr2=0x7f00a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:36.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.654+0000 7f00a79a6700 1 --2- 172.21.15.148:0/2395830180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a00f6510 0x7f00a00f6930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0090009fb0 tx=0x7f0090009b30).stop 2023-12-21T03:40:36.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.654+0000 7f00a79a6700 1 -- 172.21.15.148:0/2395830180 shutdown_connections 2023-12-21T03:40:36.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.654+0000 7f00a79a6700 1 --2- 172.21.15.148:0/2395830180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a00f6510 0x7f00a00f6930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:36.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.654+0000 7f00a79a6700 1 -- 172.21.15.148:0/2395830180 >> 172.21.15.148:0/2395830180 conn(0x7f00a00f1c70 msgr2=0x7f00a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:36.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.654+0000 7f00a79a6700 1 -- 172.21.15.148:0/2395830180 shutdown_connections 2023-12-21T03:40:36.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.654+0000 7f00a79a6700 1 -- 172.21.15.148:0/2395830180 wait complete. 2023-12-21T03:40:36.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.655+0000 7f00a79a6700 1 Processor -- start 2023-12-21T03:40:36.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.655+0000 7f00a79a6700 1 -- start start 2023-12-21T03:40:36.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.655+0000 7f00a79a6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a00f6510 0x7f00a01066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:36.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.655+0000 7f00a79a6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0106bc0 con 0x7f00a00f6510 2023-12-21T03:40:36.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.655+0000 7f00a5742700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a00f6510 0x7f00a01066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:36.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.655+0000 7f00a5742700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a00f6510 0x7f00a01066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35444/0 (socket says 172.21.15.148:35444) 2023-12-21T03:40:36.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.655+0000 7f00a5742700 1 -- 172.21.15.148:0/3756817813 learned_addr learned my addr 172.21.15.148:0/3756817813 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:36.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.655+0000 7f00a5742700 1 -- 172.21.15.148:0/3756817813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0090005040 con 0x7f00a00f6510 2023-12-21T03:40:36.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.656+0000 7f00a5742700 1 --2- 172.21.15.148:0/3756817813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a00f6510 0x7f00a01066d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0090013510 tx=0x7f009000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:36.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.656+0000 7f0095ffb700 1 -- 172.21.15.148:0/3756817813 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0090009320 con 0x7f00a00f6510 2023-12-21T03:40:36.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.656+0000 7f00a79a6700 1 -- 172.21.15.148:0/3756817813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a0106dc0 con 0x7f00a00f6510 2023-12-21T03:40:36.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.656+0000 7f0095ffb700 1 -- 172.21.15.148:0/3756817813 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0090013f00 con 0x7f00a00f6510 2023-12-21T03:40:36.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.656+0000 7f0095ffb700 1 -- 172.21.15.148:0/3756817813 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0090014d90 con 0x7f00a00f6510 2023-12-21T03:40:36.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.656+0000 7f00a79a6700 1 -- 172.21.15.148:0/3756817813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a0107250 con 0x7f00a00f6510 2023-12-21T03:40:36.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.657+0000 7f0095ffb700 1 -- 172.21.15.148:0/3756817813 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00900095f0 con 0x7f00a00f6510 2023-12-21T03:40:36.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.657+0000 7f00a79a6700 1 -- 172.21.15.148:0/3756817813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a0045ad0 con 0x7f00a00f6510 2023-12-21T03:40:36.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.657+0000 7f0095ffb700 1 --2- 172.21.15.148:0/3756817813 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f008c02ff60 0x7f008c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:36.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.657+0000 7f0095ffb700 1 -- 172.21.15.148:0/3756817813 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0090041000 con 0x7f00a00f6510 2023-12-21T03:40:36.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.657+0000 7f00a4f41700 1 --2- 172.21.15.148:0/3756817813 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f008c02ff60 0x7f008c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:36.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.658+0000 7f00a4f41700 1 --2- 172.21.15.148:0/3756817813 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f008c02ff60 0x7f008c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f009c0096a0 tx=0x7f009c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:36.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.662+0000 7f0095ffb700 1 -- 172.21.15.148:0/3756817813 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0090014420 con 0x7f00a00f6510 2023-12-21T03:40:36.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:36 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:36.410876+0000 mgr.smithi078.zidbgq (mgr.14162) 38 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:36.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.870+0000 7f00a79a6700 1 -- 172.21.15.148:0/3756817813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00a0108100 con 0x7f00a00f6510 2023-12-21T03:40:36.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.871+0000 7f0095ffb700 1 -- 172.21.15.148:0/3756817813 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f009001d7f0 con 0x7f00a00f6510 2023-12-21T03:40:36.872 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:36.872 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.872+0000 7f00a79a6700 1 -- 172.21.15.148:0/3756817813 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f008c02ff60 msgr2=0x7f008c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.872+0000 7f00a79a6700 1 --2- 172.21.15.148:0/3756817813 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f008c02ff60 0x7f008c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f009c0096a0 tx=0x7f009c006b40).stop 2023-12-21T03:40:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.872+0000 7f00a79a6700 1 -- 172.21.15.148:0/3756817813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a00f6510 msgr2=0x7f00a01066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.872+0000 7f00a79a6700 1 --2- 172.21.15.148:0/3756817813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a00f6510 0x7f00a01066d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0090013510 tx=0x7f009000b040).stop 2023-12-21T03:40:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.872+0000 7f00a79a6700 1 -- 172.21.15.148:0/3756817813 shutdown_connections 2023-12-21T03:40:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.872+0000 7f00a79a6700 1 --2- 172.21.15.148:0/3756817813 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f008c02ff60 0x7f008c032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.872+0000 7f00a79a6700 1 --2- 172.21.15.148:0/3756817813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a00f6510 0x7f00a01066d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.872+0000 7f00a79a6700 1 -- 172.21.15.148:0/3756817813 >> 172.21.15.148:0/3756817813 conn(0x7f00a00f1c70 msgr2=0x7f00a00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.873+0000 7f00a79a6700 1 -- 172.21.15.148:0/3756817813 shutdown_connections 2023-12-21T03:40:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:36.873+0000 7f00a79a6700 1 -- 172.21.15.148:0/3756817813 wait complete. 2023-12-21T03:40:36.875 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:36.871498+0000 mon.smithi078 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.148:0/3756817813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:38.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:38.411313+0000 mgr.smithi078.zidbgq (mgr.14162) 39 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:39.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:39.565 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:40.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:40.411594+0000 mgr.smithi078.zidbgq (mgr.14162) 40 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:42.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:42.411953+0000 mgr.smithi078.zidbgq (mgr.14162) 41 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:43.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.924+0000 7f39009b2700 1 -- 172.21.15.148:0/3693804223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38fc0f5410 msgr2=0x7f38fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:43.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.924+0000 7f39009b2700 1 --2- 172.21.15.148:0/3693804223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38fc0f5410 0x7f38fc0f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f38ec009fb0 tx=0x7f38ec009b30).stop 2023-12-21T03:40:43.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.924+0000 7f39009b2700 1 -- 172.21.15.148:0/3693804223 shutdown_connections 2023-12-21T03:40:43.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.924+0000 7f39009b2700 1 --2- 172.21.15.148:0/3693804223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38fc0f5410 0x7f38fc0f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:43.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.924+0000 7f39009b2700 1 -- 172.21.15.148:0/3693804223 >> 172.21.15.148:0/3693804223 conn(0x7f38fc0f0b70 msgr2=0x7f38fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:43.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.924+0000 7f39009b2700 1 -- 172.21.15.148:0/3693804223 shutdown_connections 2023-12-21T03:40:43.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.924+0000 7f39009b2700 1 -- 172.21.15.148:0/3693804223 wait complete. 2023-12-21T03:40:43.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.924+0000 7f39009b2700 1 Processor -- start 2023-12-21T03:40:43.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.924+0000 7f39009b2700 1 -- start start 2023-12-21T03:40:43.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.924+0000 7f39009b2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38fc0f5410 0x7f38fc1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:43.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.924+0000 7f39009b2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc060220 con 0x7f38fc0f5410 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.925+0000 7f38fa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38fc0f5410 0x7f38fc1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.925+0000 7f38fa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38fc0f5410 0x7f38fc1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:39012/0 (socket says 172.21.15.148:39012) 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.925+0000 7f38fa59c700 1 -- 172.21.15.148:0/3274896139 learned_addr learned my addr 172.21.15.148:0/3274896139 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.925+0000 7f38fa59c700 1 -- 172.21.15.148:0/3274896139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec005040 con 0x7f38fc0f5410 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.925+0000 7f38fa59c700 1 --2- 172.21.15.148:0/3274896139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38fc0f5410 0x7f38fc1093f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f38ec013510 tx=0x7f38ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.925+0000 7f38eaffd700 1 -- 172.21.15.148:0/3274896139 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38ec009320 con 0x7f38fc0f5410 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.925+0000 7f39009b2700 1 -- 172.21.15.148:0/3274896139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38fc060420 con 0x7f38fc0f5410 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.925+0000 7f38eaffd700 1 -- 172.21.15.148:0/3274896139 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38ec013f00 con 0x7f38fc0f5410 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.925+0000 7f38eaffd700 1 -- 172.21.15.148:0/3274896139 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38ec014d90 con 0x7f38fc0f5410 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.925+0000 7f39009b2700 1 -- 172.21.15.148:0/3274896139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38fc109aa0 con 0x7f38fc0f5410 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.926+0000 7f39009b2700 1 -- 172.21.15.148:0/3274896139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38fc059100 con 0x7f38fc0f5410 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.926+0000 7f38eaffd700 1 -- 172.21.15.148:0/3274896139 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38ec014420 con 0x7f38fc0f5410 2023-12-21T03:40:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.926+0000 7f38eaffd700 1 --2- 172.21.15.148:0/3274896139 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f38e402ff60 0x7f38e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:43.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.926+0000 7f38eaffd700 1 -- 172.21.15.148:0/3274896139 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38ec040ff0 con 0x7f38fc0f5410 2023-12-21T03:40:43.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.927+0000 7f38f9d9b700 1 --2- 172.21.15.148:0/3274896139 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f38e402ff60 0x7f38e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:43.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.927+0000 7f38f9d9b700 1 --2- 172.21.15.148:0/3274896139 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f38e402ff60 0x7f38e4032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f38f00096a0 tx=0x7f38f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:43.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:43.931+0000 7f38eaffd700 1 -- 172.21.15.148:0/3274896139 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38ec0095f0 con 0x7f38fc0f5410 2023-12-21T03:40:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.134+0000 7f39009b2700 1 -- 172.21.15.148:0/3274896139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38fc045ad0 con 0x7f38fc0f5410 2023-12-21T03:40:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.135+0000 7f38eaffd700 1 -- 172.21.15.148:0/3274896139 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f38ec01db70 con 0x7f38fc0f5410 2023-12-21T03:40:44.136 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:44.136 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.137+0000 7f39009b2700 1 -- 172.21.15.148:0/3274896139 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f38e402ff60 msgr2=0x7f38e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.137+0000 7f39009b2700 1 --2- 172.21.15.148:0/3274896139 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f38e402ff60 0x7f38e4032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f38f00096a0 tx=0x7f38f0006b40).stop 2023-12-21T03:40:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.137+0000 7f39009b2700 1 -- 172.21.15.148:0/3274896139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38fc0f5410 msgr2=0x7f38fc1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.137+0000 7f39009b2700 1 --2- 172.21.15.148:0/3274896139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38fc0f5410 0x7f38fc1093f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f38ec013510 tx=0x7f38ec00b040).stop 2023-12-21T03:40:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.137+0000 7f39009b2700 1 -- 172.21.15.148:0/3274896139 shutdown_connections 2023-12-21T03:40:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.137+0000 7f39009b2700 1 --2- 172.21.15.148:0/3274896139 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f38e402ff60 0x7f38e4032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.137+0000 7f39009b2700 1 --2- 172.21.15.148:0/3274896139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38fc0f5410 0x7f38fc1093f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.137+0000 7f39009b2700 1 -- 172.21.15.148:0/3274896139 >> 172.21.15.148:0/3274896139 conn(0x7f38fc0f0b70 msgr2=0x7f38fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.137+0000 7f39009b2700 1 -- 172.21.15.148:0/3274896139 shutdown_connections 2023-12-21T03:40:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:44.137+0000 7f39009b2700 1 -- 172.21.15.148:0/3274896139 wait complete. 2023-12-21T03:40:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:44.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:44.135747+0000 mon.smithi078 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.148:0/3274896139' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:45.526 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:45.526 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:45.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:44.412225+0000 mgr.smithi078.zidbgq (mgr.14162) 42 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:46.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.656+0000 7fa6d64f5700 1 -- 172.21.15.148:0/688360923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0060180 msgr2=0x7fa6d00f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:46.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.656+0000 7fa6d64f5700 1 --2- 172.21.15.148:0/688360923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0060180 0x7fa6d00f8f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa6cc009fb0 tx=0x7fa6cc009b30).stop 2023-12-21T03:40:46.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.656+0000 7fa6d64f5700 1 -- 172.21.15.148:0/688360923 shutdown_connections 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.656+0000 7fa6d64f5700 1 --2- 172.21.15.148:0/688360923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0060180 0x7fa6d00f8f80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.656+0000 7fa6d64f5700 1 -- 172.21.15.148:0/688360923 >> 172.21.15.148:0/688360923 conn(0x7fa6d00f09a0 msgr2=0x7fa6d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.656+0000 7fa6d64f5700 1 -- 172.21.15.148:0/688360923 shutdown_connections 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.656+0000 7fa6d64f5700 1 -- 172.21.15.148:0/688360923 wait complete. 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.657+0000 7fa6d64f5700 1 Processor -- start 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.657+0000 7fa6d64f5700 1 -- start start 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.657+0000 7fa6d64f5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0060180 0x7fa6d0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.657+0000 7fa6d64f5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d01069f0 con 0x7fa6d0060180 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.657+0000 7fa6d54f3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0060180 0x7fa6d0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.657+0000 7fa6d54f3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0060180 0x7fa6d0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:39030/0 (socket says 172.21.15.148:39030) 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.657+0000 7fa6d54f3700 1 -- 172.21.15.148:0/3122080963 learned_addr learned my addr 172.21.15.148:0/3122080963 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:46.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.657+0000 7fa6d54f3700 1 -- 172.21.15.148:0/3122080963 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc005040 con 0x7fa6d0060180 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.657+0000 7fa6d54f3700 1 --2- 172.21.15.148:0/3122080963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0060180 0x7fa6d0106500 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fa6cc014000 tx=0x7fa6cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.658+0000 7fa6c5ffb700 1 -- 172.21.15.148:0/3122080963 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6cc0095f0 con 0x7fa6d0060180 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.658+0000 7fa6d64f5700 1 -- 172.21.15.148:0/3122080963 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d0106bf0 con 0x7fa6d0060180 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.658+0000 7fa6c5ffb700 1 -- 172.21.15.148:0/3122080963 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6cc007e70 con 0x7fa6d0060180 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.658+0000 7fa6c5ffb700 1 -- 172.21.15.148:0/3122080963 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6cc014ea0 con 0x7fa6d0060180 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.658+0000 7fa6d64f5700 1 -- 172.21.15.148:0/3122080963 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d0107080 con 0x7fa6d0060180 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.659+0000 7fa6c5ffb700 1 -- 172.21.15.148:0/3122080963 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6cc025070 con 0x7fa6d0060180 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.659+0000 7fa6d64f5700 1 -- 172.21.15.148:0/3122080963 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d0045ad0 con 0x7fa6d0060180 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.659+0000 7fa6c5ffb700 1 --2- 172.21.15.148:0/3122080963 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa6bc02ffb0 0x7fa6bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.659+0000 7fa6c5ffb700 1 -- 172.21.15.148:0/3122080963 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6cc0409e0 con 0x7fa6d0060180 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.659+0000 7fa6d4cf2700 1 --2- 172.21.15.148:0/3122080963 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa6bc02ffb0 0x7fa6bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:46.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.659+0000 7fa6d4cf2700 1 --2- 172.21.15.148:0/3122080963 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa6bc02ffb0 0x7fa6bc032470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fa6c00096a0 tx=0x7fa6c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:46.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.664+0000 7fa6c5ffb700 1 -- 172.21.15.148:0/3122080963 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6cc0079c0 con 0x7fa6d0060180 2023-12-21T03:40:46.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:46 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:46.412476+0000 mgr.smithi078.zidbgq (mgr.14162) 43 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:46.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.872+0000 7fa6d64f5700 1 -- 172.21.15.148:0/3122080963 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6d0107f30 con 0x7fa6d0060180 2023-12-21T03:40:46.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.873+0000 7fa6c5ffb700 1 -- 172.21.15.148:0/3122080963 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6cc00f300 con 0x7fa6d0060180 2023-12-21T03:40:46.874 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:46.874 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:46.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.874+0000 7fa6d64f5700 1 -- 172.21.15.148:0/3122080963 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa6bc02ffb0 msgr2=0x7fa6bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:46.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.874+0000 7fa6d64f5700 1 --2- 172.21.15.148:0/3122080963 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa6bc02ffb0 0x7fa6bc032470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fa6c00096a0 tx=0x7fa6c0006b40).stop 2023-12-21T03:40:46.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.874+0000 7fa6d64f5700 1 -- 172.21.15.148:0/3122080963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0060180 msgr2=0x7fa6d0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:46.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.874+0000 7fa6d64f5700 1 --2- 172.21.15.148:0/3122080963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0060180 0x7fa6d0106500 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fa6cc014000 tx=0x7fa6cc00b040).stop 2023-12-21T03:40:46.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.875+0000 7fa6d64f5700 1 -- 172.21.15.148:0/3122080963 shutdown_connections 2023-12-21T03:40:46.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.875+0000 7fa6d64f5700 1 --2- 172.21.15.148:0/3122080963 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa6bc02ffb0 0x7fa6bc032470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:46.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.875+0000 7fa6d64f5700 1 --2- 172.21.15.148:0/3122080963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0060180 0x7fa6d0106500 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:46.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.875+0000 7fa6d64f5700 1 -- 172.21.15.148:0/3122080963 >> 172.21.15.148:0/3122080963 conn(0x7fa6d00f09a0 msgr2=0x7fa6d00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:46.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.875+0000 7fa6d64f5700 1 -- 172.21.15.148:0/3122080963 shutdown_connections 2023-12-21T03:40:46.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:46.875+0000 7fa6d64f5700 1 -- 172.21.15.148:0/3122080963 wait complete. 2023-12-21T03:40:46.877 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:46.873612+0000 mon.smithi078 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.148:0/3122080963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:48.307 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:48.307 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:48.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:48.412676+0000 mgr.smithi078.zidbgq (mgr.14162) 44 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.453+0000 7f2dbaf33700 1 -- 172.21.15.148:0/442732212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db40f6510 msgr2=0x7f2db40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.453+0000 7f2dbaf33700 1 --2- 172.21.15.148:0/442732212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db40f6510 0x7f2db40f6930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2da8009fb0 tx=0x7f2da8009b30).stop 2023-12-21T03:40:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.453+0000 7f2dbaf33700 1 -- 172.21.15.148:0/442732212 shutdown_connections 2023-12-21T03:40:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.453+0000 7f2dbaf33700 1 --2- 172.21.15.148:0/442732212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db40f6510 0x7f2db40f6930 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.453+0000 7f2dbaf33700 1 -- 172.21.15.148:0/442732212 >> 172.21.15.148:0/442732212 conn(0x7f2db40f1c70 msgr2=0x7f2db40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.453+0000 7f2dbaf33700 1 -- 172.21.15.148:0/442732212 shutdown_connections 2023-12-21T03:40:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.453+0000 7f2dbaf33700 1 -- 172.21.15.148:0/442732212 wait complete. 2023-12-21T03:40:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.454+0000 7f2dbaf33700 1 Processor -- start 2023-12-21T03:40:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.454+0000 7f2dbaf33700 1 -- start start 2023-12-21T03:40:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.454+0000 7f2dbaf33700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db406b960 0x7f2db4069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.454+0000 7f2dbaf33700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db406bd80 con 0x7f2db406b960 2023-12-21T03:40:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.455+0000 7f2db8ccf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db406b960 0x7f2db4069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.455+0000 7f2db8ccf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db406b960 0x7f2db4069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:55422/0 (socket says 172.21.15.148:55422) 2023-12-21T03:40:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.455+0000 7f2db8ccf700 1 -- 172.21.15.148:0/297053972 learned_addr learned my addr 172.21.15.148:0/297053972 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.455+0000 7f2db8ccf700 1 -- 172.21.15.148:0/297053972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da8005040 con 0x7f2db406b960 2023-12-21T03:40:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.455+0000 7f2db8ccf700 1 --2- 172.21.15.148:0/297053972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db406b960 0x7f2db4069e20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2da8009f80 tx=0x7f2da8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.455+0000 7f2db17fa700 1 -- 172.21.15.148:0/297053972 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2da80136c0 con 0x7f2db406b960 2023-12-21T03:40:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.455+0000 7f2dbaf33700 1 -- 172.21.15.148:0/297053972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db406a310 con 0x7f2db406b960 2023-12-21T03:40:49.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.455+0000 7f2db17fa700 1 -- 172.21.15.148:0/297053972 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2da8013e70 con 0x7f2db406b960 2023-12-21T03:40:49.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.455+0000 7f2db17fa700 1 -- 172.21.15.148:0/297053972 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2da8014420 con 0x7f2db406b960 2023-12-21T03:40:49.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.455+0000 7f2dbaf33700 1 -- 172.21.15.148:0/297053972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db406a800 con 0x7f2db406b960 2023-12-21T03:40:49.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.456+0000 7f2db17fa700 1 -- 172.21.15.148:0/297053972 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2da8025060 con 0x7f2db406b960 2023-12-21T03:40:49.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.456+0000 7f2dbaf33700 1 -- 172.21.15.148:0/297053972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db4045ad0 con 0x7f2db406b960 2023-12-21T03:40:49.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.456+0000 7f2db17fa700 1 --2- 172.21.15.148:0/297053972 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2da402ff60 0x7f2da4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:49.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.456+0000 7f2db17fa700 1 -- 172.21.15.148:0/297053972 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2da8040ec0 con 0x7f2db406b960 2023-12-21T03:40:49.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.457+0000 7f2db3fff700 1 --2- 172.21.15.148:0/297053972 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2da402ff60 0x7f2da4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:49.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.457+0000 7f2db3fff700 1 --2- 172.21.15.148:0/297053972 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2da402ff60 0x7f2da4032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f2da00096a0 tx=0x7f2da0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:49.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.461+0000 7f2db17fa700 1 -- 172.21.15.148:0/297053972 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2da801cd10 con 0x7f2db406b960 2023-12-21T03:40:49.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.666+0000 7f2dbaf33700 1 -- 172.21.15.148:0/297053972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2db410ca40 con 0x7f2db406b960 2023-12-21T03:40:49.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.667+0000 7f2db17fa700 1 -- 172.21.15.148:0/297053972 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2da802d2d0 con 0x7f2db406b960 2023-12-21T03:40:49.668 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:49.668 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:49.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.668+0000 7f2dbaf33700 1 -- 172.21.15.148:0/297053972 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2da402ff60 msgr2=0x7f2da4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.668+0000 7f2dbaf33700 1 --2- 172.21.15.148:0/297053972 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2da402ff60 0x7f2da4032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f2da00096a0 tx=0x7f2da0006b40).stop 2023-12-21T03:40:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.668+0000 7f2dbaf33700 1 -- 172.21.15.148:0/297053972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db406b960 msgr2=0x7f2db4069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.668+0000 7f2dbaf33700 1 --2- 172.21.15.148:0/297053972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db406b960 0x7f2db4069e20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2da8009f80 tx=0x7f2da8009b30).stop 2023-12-21T03:40:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.668+0000 7f2dbaf33700 1 -- 172.21.15.148:0/297053972 shutdown_connections 2023-12-21T03:40:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.668+0000 7f2dbaf33700 1 --2- 172.21.15.148:0/297053972 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2da402ff60 0x7f2da4032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.668+0000 7f2dbaf33700 1 --2- 172.21.15.148:0/297053972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db406b960 0x7f2db4069e20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.668+0000 7f2dbaf33700 1 -- 172.21.15.148:0/297053972 >> 172.21.15.148:0/297053972 conn(0x7f2db40f1c70 msgr2=0x7f2db40f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.668+0000 7f2dbaf33700 1 -- 172.21.15.148:0/297053972 shutdown_connections 2023-12-21T03:40:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:49.668+0000 7f2dbaf33700 1 -- 172.21.15.148:0/297053972 wait complete. 2023-12-21T03:40:49.671 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:50.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:49.667648+0000 mon.smithi078 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.148:0/297053972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:51.080 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:51.080 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:51.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:50.413047+0000 mgr.smithi078.zidbgq (mgr.14162) 45 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:52.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.244+0000 7efe27c1e700 1 -- 172.21.15.148:0/1593185052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe200f47e0 msgr2=0x7efe200f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.244+0000 7efe27c1e700 1 --2- 172.21.15.148:0/1593185052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe200f47e0 0x7efe200f4c00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7efe10009fb0 tx=0x7efe10009b30).stop 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.244+0000 7efe27c1e700 1 -- 172.21.15.148:0/1593185052 shutdown_connections 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.244+0000 7efe27c1e700 1 --2- 172.21.15.148:0/1593185052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe200f47e0 0x7efe200f4c00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.244+0000 7efe27c1e700 1 -- 172.21.15.148:0/1593185052 >> 172.21.15.148:0/1593185052 conn(0x7efe200f0260 msgr2=0x7efe200f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.244+0000 7efe27c1e700 1 -- 172.21.15.148:0/1593185052 shutdown_connections 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.244+0000 7efe27c1e700 1 -- 172.21.15.148:0/1593185052 wait complete. 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.245+0000 7efe27c1e700 1 Processor -- start 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.245+0000 7efe27c1e700 1 -- start start 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.245+0000 7efe27c1e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe200f47e0 0x7efe200f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.245+0000 7efe27c1e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe200f5850 con 0x7efe200f47e0 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.245+0000 7efe259ba700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe200f47e0 0x7efe200f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:52.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.245+0000 7efe259ba700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe200f47e0 0x7efe200f73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:55452/0 (socket says 172.21.15.148:55452) 2023-12-21T03:40:52.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.245+0000 7efe259ba700 1 -- 172.21.15.148:0/2670467881 learned_addr learned my addr 172.21.15.148:0/2670467881 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:52.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.246+0000 7efe259ba700 1 -- 172.21.15.148:0/2670467881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe10005040 con 0x7efe200f47e0 2023-12-21T03:40:52.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.246+0000 7efe259ba700 1 --2- 172.21.15.148:0/2670467881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe200f47e0 0x7efe200f73d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7efe10013510 tx=0x7efe1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:52.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.246+0000 7efe167fc700 1 -- 172.21.15.148:0/2670467881 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe10009320 con 0x7efe200f47e0 2023-12-21T03:40:52.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.246+0000 7efe27c1e700 1 -- 172.21.15.148:0/2670467881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe200f59f0 con 0x7efe200f47e0 2023-12-21T03:40:52.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.246+0000 7efe27c1e700 1 -- 172.21.15.148:0/2670467881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe200f5e80 con 0x7efe200f47e0 2023-12-21T03:40:52.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.246+0000 7efe167fc700 1 -- 172.21.15.148:0/2670467881 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe10013f00 con 0x7efe200f47e0 2023-12-21T03:40:52.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.246+0000 7efe167fc700 1 -- 172.21.15.148:0/2670467881 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe10014d90 con 0x7efe200f47e0 2023-12-21T03:40:52.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.247+0000 7efe167fc700 1 -- 172.21.15.148:0/2670467881 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe100263a0 con 0x7efe200f47e0 2023-12-21T03:40:52.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.247+0000 7efe167fc700 1 --2- 172.21.15.148:0/2670467881 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efe0c02ffb0 0x7efe0c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:52.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.247+0000 7efe27c1e700 1 -- 172.21.15.148:0/2670467881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe08004fa0 con 0x7efe200f47e0 2023-12-21T03:40:52.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.247+0000 7efe167fc700 1 -- 172.21.15.148:0/2670467881 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe10041060 con 0x7efe200f47e0 2023-12-21T03:40:52.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.247+0000 7efe251b9700 1 --2- 172.21.15.148:0/2670467881 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efe0c02ffb0 0x7efe0c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:52.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.248+0000 7efe251b9700 1 --2- 172.21.15.148:0/2670467881 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efe0c02ffb0 0x7efe0c032470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7efe1c0096a0 tx=0x7efe1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:52.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.252+0000 7efe167fc700 1 -- 172.21.15.148:0/2670467881 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe1001d380 con 0x7efe200f47e0 2023-12-21T03:40:52.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.459+0000 7efe27c1e700 1 -- 172.21.15.148:0/2670467881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe08005670 con 0x7efe200f47e0 2023-12-21T03:40:52.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.460+0000 7efe167fc700 1 -- 172.21.15.148:0/2670467881 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe10012330 con 0x7efe200f47e0 2023-12-21T03:40:52.461 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:52.461 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:52.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.461+0000 7efe27c1e700 1 -- 172.21.15.148:0/2670467881 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efe0c02ffb0 msgr2=0x7efe0c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:52.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.461+0000 7efe27c1e700 1 --2- 172.21.15.148:0/2670467881 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efe0c02ffb0 0x7efe0c032470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7efe1c0096a0 tx=0x7efe1c006b40).stop 2023-12-21T03:40:52.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.461+0000 7efe27c1e700 1 -- 172.21.15.148:0/2670467881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe200f47e0 msgr2=0x7efe200f73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:52.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.461+0000 7efe27c1e700 1 --2- 172.21.15.148:0/2670467881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe200f47e0 0x7efe200f73d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7efe10013510 tx=0x7efe1000b040).stop 2023-12-21T03:40:52.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.461+0000 7efe27c1e700 1 -- 172.21.15.148:0/2670467881 shutdown_connections 2023-12-21T03:40:52.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.461+0000 7efe27c1e700 1 --2- 172.21.15.148:0/2670467881 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efe0c02ffb0 0x7efe0c032470 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:52.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.461+0000 7efe27c1e700 1 --2- 172.21.15.148:0/2670467881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe200f47e0 0x7efe200f73d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:52.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.461+0000 7efe27c1e700 1 -- 172.21.15.148:0/2670467881 >> 172.21.15.148:0/2670467881 conn(0x7efe200f0260 msgr2=0x7efe200f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:52.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.461+0000 7efe27c1e700 1 -- 172.21.15.148:0/2670467881 shutdown_connections 2023-12-21T03:40:52.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:52.461+0000 7efe27c1e700 1 -- 172.21.15.148:0/2670467881 wait complete. 2023-12-21T03:40:52.463 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:52.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:52.413419+0000 mgr.smithi078.zidbgq (mgr.14162) 46 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:52.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:52.460617+0000 mon.smithi078 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.148:0/2670467881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:53.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:53.889 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:55.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.054+0000 7f3a51a3b700 1 -- 172.21.15.148:0/3251892159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a4c06b920 msgr2=0x7f3a4c069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:55.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.054+0000 7f3a51a3b700 1 --2- 172.21.15.148:0/3251892159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a4c06b920 0x7f3a4c069ec0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f3a40009fb0 tx=0x7f3a40009b30).stop 2023-12-21T03:40:55.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.054+0000 7f3a51a3b700 1 -- 172.21.15.148:0/3251892159 shutdown_connections 2023-12-21T03:40:55.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.054+0000 7f3a51a3b700 1 --2- 172.21.15.148:0/3251892159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a4c06b920 0x7f3a4c069ec0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:55.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.054+0000 7f3a51a3b700 1 -- 172.21.15.148:0/3251892159 >> 172.21.15.148:0/3251892159 conn(0x7f3a4c0f1ac0 msgr2=0x7f3a4c0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:55.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.054+0000 7f3a51a3b700 1 -- 172.21.15.148:0/3251892159 shutdown_connections 2023-12-21T03:40:55.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.054+0000 7f3a51a3b700 1 -- 172.21.15.148:0/3251892159 wait complete. 2023-12-21T03:40:55.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.054+0000 7f3a51a3b700 1 Processor -- start 2023-12-21T03:40:55.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.054+0000 7f3a51a3b700 1 -- start start 2023-12-21T03:40:55.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.055+0000 7f3a51a3b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a4c06b920 0x7f3a4c1086e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:55.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.055+0000 7f3a51a3b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c108bd0 con 0x7f3a4c06b920 2023-12-21T03:40:55.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.055+0000 7f3a50a39700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a4c06b920 0x7f3a4c1086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:55.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.055+0000 7f3a50a39700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a4c06b920 0x7f3a4c1086e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:55464/0 (socket says 172.21.15.148:55464) 2023-12-21T03:40:55.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.055+0000 7f3a50a39700 1 -- 172.21.15.148:0/1121579031 learned_addr learned my addr 172.21.15.148:0/1121579031 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:55.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.055+0000 7f3a50a39700 1 -- 172.21.15.148:0/1121579031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a40005040 con 0x7f3a4c06b920 2023-12-21T03:40:55.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.055+0000 7f3a50a39700 1 --2- 172.21.15.148:0/1121579031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a4c06b920 0x7f3a4c1086e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f3a40014000 tx=0x7f3a4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:55.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.055+0000 7f3a497fa700 1 -- 172.21.15.148:0/1121579031 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a400095f0 con 0x7f3a4c06b920 2023-12-21T03:40:55.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.055+0000 7f3a497fa700 1 -- 172.21.15.148:0/1121579031 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a40007e70 con 0x7f3a4c06b920 2023-12-21T03:40:55.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.055+0000 7f3a51a3b700 1 -- 172.21.15.148:0/1121579031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4c108dd0 con 0x7f3a4c06b920 2023-12-21T03:40:55.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.056+0000 7f3a497fa700 1 -- 172.21.15.148:0/1121579031 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a40014ea0 con 0x7f3a4c06b920 2023-12-21T03:40:55.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.056+0000 7f3a51a3b700 1 -- 172.21.15.148:0/1121579031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c109260 con 0x7f3a4c06b920 2023-12-21T03:40:55.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.056+0000 7f3a497fa700 1 -- 172.21.15.148:0/1121579031 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a40025070 con 0x7f3a4c06b920 2023-12-21T03:40:55.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.056+0000 7f3a51a3b700 1 -- 172.21.15.148:0/1121579031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4c045ad0 con 0x7f3a4c06b920 2023-12-21T03:40:55.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.056+0000 7f3a497fa700 1 --2- 172.21.15.148:0/1121579031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3a3c02ff60 0x7f3a3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:55.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.057+0000 7f3a497fa700 1 -- 172.21.15.148:0/1121579031 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a400409b0 con 0x7f3a4c06b920 2023-12-21T03:40:55.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.057+0000 7f3a4bfff700 1 --2- 172.21.15.148:0/1121579031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3a3c02ff60 0x7f3a3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:55.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.057+0000 7f3a4bfff700 1 --2- 172.21.15.148:0/1121579031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3a3c02ff60 0x7f3a3c032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f3a380096a0 tx=0x7f3a38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:55.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.062+0000 7f3a497fa700 1 -- 172.21.15.148:0/1121579031 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a4000f4e0 con 0x7f3a4c06b920 2023-12-21T03:40:55.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.268+0000 7f3a51a3b700 1 -- 172.21.15.148:0/1121579031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a4c10a110 con 0x7f3a4c06b920 2023-12-21T03:40:55.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.269+0000 7f3a497fa700 1 -- 172.21.15.148:0/1121579031 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a4000f300 con 0x7f3a4c06b920 2023-12-21T03:40:55.271 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:55.271 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:55.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.271+0000 7f3a51a3b700 1 -- 172.21.15.148:0/1121579031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3a3c02ff60 msgr2=0x7f3a3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:55.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.271+0000 7f3a51a3b700 1 --2- 172.21.15.148:0/1121579031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3a3c02ff60 0x7f3a3c032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f3a380096a0 tx=0x7f3a38006b40).stop 2023-12-21T03:40:55.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.271+0000 7f3a51a3b700 1 -- 172.21.15.148:0/1121579031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a4c06b920 msgr2=0x7f3a4c1086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:55.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.271+0000 7f3a51a3b700 1 --2- 172.21.15.148:0/1121579031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a4c06b920 0x7f3a4c1086e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f3a40014000 tx=0x7f3a4000b040).stop 2023-12-21T03:40:55.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.271+0000 7f3a51a3b700 1 -- 172.21.15.148:0/1121579031 shutdown_connections 2023-12-21T03:40:55.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.271+0000 7f3a51a3b700 1 --2- 172.21.15.148:0/1121579031 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3a3c02ff60 0x7f3a3c032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:55.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.271+0000 7f3a51a3b700 1 --2- 172.21.15.148:0/1121579031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a4c06b920 0x7f3a4c1086e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:55.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.271+0000 7f3a51a3b700 1 -- 172.21.15.148:0/1121579031 >> 172.21.15.148:0/1121579031 conn(0x7f3a4c0f1ac0 msgr2=0x7f3a4c0f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:55.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.271+0000 7f3a51a3b700 1 -- 172.21.15.148:0/1121579031 shutdown_connections 2023-12-21T03:40:55.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:55.271+0000 7f3a51a3b700 1 -- 172.21.15.148:0/1121579031 wait complete. 2023-12-21T03:40:55.274 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:55.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:54.413732+0000 mgr.smithi078.zidbgq (mgr.14162) 47 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:56.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:55.270028+0000 mon.smithi078 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.148:0/1121579031' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:56.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:56.654 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:57.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:56.414075+0000 mgr.smithi078.zidbgq (mgr.14162) 48 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:40:57.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.834+0000 7fc7a9147700 1 -- 172.21.15.148:0/1382049343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a40f5410 msgr2=0x7fc7a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:57.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.834+0000 7fc7a9147700 1 --2- 172.21.15.148:0/1382049343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a40f5410 0x7fc7a40f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc794009fb0 tx=0x7fc794009b30).stop 2023-12-21T03:40:57.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.834+0000 7fc7a9147700 1 -- 172.21.15.148:0/1382049343 shutdown_connections 2023-12-21T03:40:57.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.834+0000 7fc7a9147700 1 --2- 172.21.15.148:0/1382049343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a40f5410 0x7fc7a40f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:57.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.834+0000 7fc7a9147700 1 -- 172.21.15.148:0/1382049343 >> 172.21.15.148:0/1382049343 conn(0x7fc7a40f0b70 msgr2=0x7fc7a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:57.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.834+0000 7fc7a9147700 1 -- 172.21.15.148:0/1382049343 shutdown_connections 2023-12-21T03:40:57.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.834+0000 7fc7a9147700 1 -- 172.21.15.148:0/1382049343 wait complete. 2023-12-21T03:40:57.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.835+0000 7fc7a9147700 1 Processor -- start 2023-12-21T03:40:57.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.835+0000 7fc7a9147700 1 -- start start 2023-12-21T03:40:57.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.835+0000 7fc7a9147700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a4060240 0x7fc7a4109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:57.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.835+0000 7fc7a9147700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a4109a30 con 0x7fc7a4060240 2023-12-21T03:40:57.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.836+0000 7fc7a2d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a4060240 0x7fc7a4109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:57.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.836+0000 7fc7a2d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a4060240 0x7fc7a4109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:55480/0 (socket says 172.21.15.148:55480) 2023-12-21T03:40:57.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.836+0000 7fc7a2d9d700 1 -- 172.21.15.148:0/1231189009 learned_addr learned my addr 172.21.15.148:0/1231189009 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:40:57.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.836+0000 7fc7a2d9d700 1 -- 172.21.15.148:0/1231189009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc794005040 con 0x7fc7a4060240 2023-12-21T03:40:57.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.836+0000 7fc7a2d9d700 1 --2- 172.21.15.148:0/1231189009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a4060240 0x7fc7a4109540 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc794009f80 tx=0x7fc794009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:57.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.836+0000 7fc78b7fe700 1 -- 172.21.15.148:0/1231189009 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7940136c0 con 0x7fc7a4060240 2023-12-21T03:40:57.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.836+0000 7fc7a9147700 1 -- 172.21.15.148:0/1231189009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a4109b70 con 0x7fc7a4060240 2023-12-21T03:40:57.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.836+0000 7fc78b7fe700 1 -- 172.21.15.148:0/1231189009 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc794013e70 con 0x7fc7a4060240 2023-12-21T03:40:57.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.836+0000 7fc78b7fe700 1 -- 172.21.15.148:0/1231189009 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc794014420 con 0x7fc7a4060240 2023-12-21T03:40:57.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.836+0000 7fc7a9147700 1 -- 172.21.15.148:0/1231189009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a410a000 con 0x7fc7a4060240 2023-12-21T03:40:57.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.837+0000 7fc7a9147700 1 -- 172.21.15.148:0/1231189009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a4045ad0 con 0x7fc7a4060240 2023-12-21T03:40:57.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.837+0000 7fc78b7fe700 1 -- 172.21.15.148:0/1231189009 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc794025060 con 0x7fc7a4060240 2023-12-21T03:40:57.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.837+0000 7fc78b7fe700 1 --2- 172.21.15.148:0/1231189009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc78c02ff60 0x7fc78c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:40:57.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.837+0000 7fc78b7fe700 1 -- 172.21.15.148:0/1231189009 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc794040ec0 con 0x7fc7a4060240 2023-12-21T03:40:57.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.838+0000 7fc7a259c700 1 --2- 172.21.15.148:0/1231189009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc78c02ff60 0x7fc78c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:40:57.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.838+0000 7fc7a259c700 1 --2- 172.21.15.148:0/1231189009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc78c02ff60 0x7fc78c032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fc7980096a0 tx=0x7fc798006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:40:57.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:57.842+0000 7fc78b7fe700 1 -- 172.21.15.148:0/1231189009 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc79401d3a0 con 0x7fc7a4060240 2023-12-21T03:40:58.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.047+0000 7fc7a9147700 1 -- 172.21.15.148:0/1231189009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7a40fae60 con 0x7fc7a4060240 2023-12-21T03:40:58.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.048+0000 7fc78b7fe700 1 -- 172.21.15.148:0/1231189009 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc79402d2d0 con 0x7fc7a4060240 2023-12-21T03:40:58.050 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:40:58.050 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:40:58.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.049+0000 7fc7a9147700 1 -- 172.21.15.148:0/1231189009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc78c02ff60 msgr2=0x7fc78c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:58.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.049+0000 7fc7a9147700 1 --2- 172.21.15.148:0/1231189009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc78c02ff60 0x7fc78c032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fc7980096a0 tx=0x7fc798006b40).stop 2023-12-21T03:40:58.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.049+0000 7fc7a9147700 1 -- 172.21.15.148:0/1231189009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a4060240 msgr2=0x7fc7a4109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:40:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.049+0000 7fc7a9147700 1 --2- 172.21.15.148:0/1231189009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a4060240 0x7fc7a4109540 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc794009f80 tx=0x7fc794009b30).stop 2023-12-21T03:40:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.049+0000 7fc7a9147700 1 -- 172.21.15.148:0/1231189009 shutdown_connections 2023-12-21T03:40:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.049+0000 7fc7a9147700 1 --2- 172.21.15.148:0/1231189009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc78c02ff60 0x7fc78c032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.049+0000 7fc7a9147700 1 --2- 172.21.15.148:0/1231189009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a4060240 0x7fc7a4109540 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:40:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.049+0000 7fc7a9147700 1 -- 172.21.15.148:0/1231189009 >> 172.21.15.148:0/1231189009 conn(0x7fc7a40f0b70 msgr2=0x7fc7a40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:40:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.049+0000 7fc7a9147700 1 -- 172.21.15.148:0/1231189009 shutdown_connections 2023-12-21T03:40:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:40:58.049+0000 7fc7a9147700 1 -- 172.21.15.148:0/1231189009 wait complete. 2023-12-21T03:40:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:40:58.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:58 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:40:58.049611+0000 mon.smithi078 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.148:0/1231189009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:40:59.459 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:40:59.459 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:40:59.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:40:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:40:58.414314+0000 mgr.smithi078.zidbgq (mgr.14162) 49 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:00.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.648+0000 7fde97475700 1 -- 172.21.15.148:0/2250895057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde900f5410 msgr2=0x7fde900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:00.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.648+0000 7fde97475700 1 --2- 172.21.15.148:0/2250895057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde900f5410 0x7fde900f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fde8c009fb0 tx=0x7fde8c009b30).stop 2023-12-21T03:41:00.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.648+0000 7fde97475700 1 -- 172.21.15.148:0/2250895057 shutdown_connections 2023-12-21T03:41:00.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.648+0000 7fde97475700 1 --2- 172.21.15.148:0/2250895057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde900f5410 0x7fde900f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:00.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.648+0000 7fde97475700 1 -- 172.21.15.148:0/2250895057 >> 172.21.15.148:0/2250895057 conn(0x7fde900f0b70 msgr2=0x7fde900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:00.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.648+0000 7fde97475700 1 -- 172.21.15.148:0/2250895057 shutdown_connections 2023-12-21T03:41:00.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.648+0000 7fde97475700 1 -- 172.21.15.148:0/2250895057 wait complete. 2023-12-21T03:41:00.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.648+0000 7fde97475700 1 Processor -- start 2023-12-21T03:41:00.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde97475700 1 -- start start 2023-12-21T03:41:00.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde97475700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde900f5410 0x7fde90106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:00.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde97475700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde90106c00 con 0x7fde900f5410 2023-12-21T03:41:00.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde95211700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde900f5410 0x7fde90106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:00.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde95211700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde900f5410 0x7fde90106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:48600/0 (socket says 172.21.15.148:48600) 2023-12-21T03:41:00.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde95211700 1 -- 172.21.15.148:0/2562416516 learned_addr learned my addr 172.21.15.148:0/2562416516 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:00.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde95211700 1 -- 172.21.15.148:0/2562416516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c005040 con 0x7fde900f5410 2023-12-21T03:41:00.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde95211700 1 --2- 172.21.15.148:0/2562416516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde900f5410 0x7fde90106710 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fde8c013510 tx=0x7fde8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:00.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde85ffb700 1 -- 172.21.15.148:0/2562416516 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde8c009320 con 0x7fde900f5410 2023-12-21T03:41:00.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde97475700 1 -- 172.21.15.148:0/2562416516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde90106e00 con 0x7fde900f5410 2023-12-21T03:41:00.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde85ffb700 1 -- 172.21.15.148:0/2562416516 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde8c013f00 con 0x7fde900f5410 2023-12-21T03:41:00.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.649+0000 7fde85ffb700 1 -- 172.21.15.148:0/2562416516 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde8c014d90 con 0x7fde900f5410 2023-12-21T03:41:00.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.650+0000 7fde97475700 1 -- 172.21.15.148:0/2562416516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde90107290 con 0x7fde900f5410 2023-12-21T03:41:00.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.651+0000 7fde85ffb700 1 -- 172.21.15.148:0/2562416516 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde8c0095f0 con 0x7fde900f5410 2023-12-21T03:41:00.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.651+0000 7fde97475700 1 -- 172.21.15.148:0/2562416516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde90045ad0 con 0x7fde900f5410 2023-12-21T03:41:00.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.651+0000 7fde85ffb700 1 --2- 172.21.15.148:0/2562416516 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fde7c02ff60 0x7fde7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:00.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.651+0000 7fde85ffb700 1 -- 172.21.15.148:0/2562416516 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde8c041000 con 0x7fde900f5410 2023-12-21T03:41:00.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.651+0000 7fde94a10700 1 --2- 172.21.15.148:0/2562416516 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fde7c02ff60 0x7fde7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:00.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.651+0000 7fde94a10700 1 --2- 172.21.15.148:0/2562416516 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fde7c02ff60 0x7fde7c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fde800096a0 tx=0x7fde80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:00.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.656+0000 7fde85ffb700 1 -- 172.21.15.148:0/2562416516 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde8c01d7f0 con 0x7fde900f5410 2023-12-21T03:41:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:00.414540+0000 mgr.smithi078.zidbgq (mgr.14162) 50 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:00.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.863+0000 7fde97475700 1 -- 172.21.15.148:0/2562416516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde900fae60 con 0x7fde900f5410 2023-12-21T03:41:00.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.864+0000 7fde85ffb700 1 -- 172.21.15.148:0/2562416516 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fde8c010230 con 0x7fde900f5410 2023-12-21T03:41:00.866 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:00.867 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:00.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.866+0000 7fde97475700 1 -- 172.21.15.148:0/2562416516 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fde7c02ff60 msgr2=0x7fde7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:00.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.866+0000 7fde97475700 1 --2- 172.21.15.148:0/2562416516 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fde7c02ff60 0x7fde7c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fde800096a0 tx=0x7fde80006b40).stop 2023-12-21T03:41:00.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.866+0000 7fde97475700 1 -- 172.21.15.148:0/2562416516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde900f5410 msgr2=0x7fde90106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:00.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.866+0000 7fde97475700 1 --2- 172.21.15.148:0/2562416516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde900f5410 0x7fde90106710 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fde8c013510 tx=0x7fde8c00b040).stop 2023-12-21T03:41:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.866+0000 7fde97475700 1 -- 172.21.15.148:0/2562416516 shutdown_connections 2023-12-21T03:41:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.866+0000 7fde97475700 1 --2- 172.21.15.148:0/2562416516 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fde7c02ff60 0x7fde7c032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.866+0000 7fde97475700 1 --2- 172.21.15.148:0/2562416516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde900f5410 0x7fde90106710 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.866+0000 7fde97475700 1 -- 172.21.15.148:0/2562416516 >> 172.21.15.148:0/2562416516 conn(0x7fde900f0b70 msgr2=0x7fde9005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.866+0000 7fde97475700 1 -- 172.21.15.148:0/2562416516 shutdown_connections 2023-12-21T03:41:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:00.866+0000 7fde97475700 1 -- 172.21.15.148:0/2562416516 wait complete. 2023-12-21T03:41:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:00.866138+0000 mon.smithi078 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.148:0/2562416516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:02.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:02.269 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:02.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:02.414894+0000 mgr.smithi078.zidbgq (mgr.14162) 51 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:03.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.423+0000 7f25e0f3e700 1 -- 172.21.15.148:0/4256419718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25dc0f5410 msgr2=0x7f25dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:03.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.423+0000 7f25e0f3e700 1 --2- 172.21.15.148:0/4256419718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25dc0f5410 0x7f25dc0f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f25cc009fb0 tx=0x7f25cc009b30).stop 2023-12-21T03:41:03.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.423+0000 7f25e0f3e700 1 -- 172.21.15.148:0/4256419718 shutdown_connections 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.423+0000 7f25e0f3e700 1 --2- 172.21.15.148:0/4256419718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25dc0f5410 0x7f25dc0f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.423+0000 7f25e0f3e700 1 -- 172.21.15.148:0/4256419718 >> 172.21.15.148:0/4256419718 conn(0x7f25dc0f0b70 msgr2=0x7f25dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.423+0000 7f25e0f3e700 1 -- 172.21.15.148:0/4256419718 shutdown_connections 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.423+0000 7f25e0f3e700 1 -- 172.21.15.148:0/4256419718 wait complete. 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.423+0000 7f25e0f3e700 1 Processor -- start 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.423+0000 7f25e0f3e700 1 -- start start 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25e0f3e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25dc0f5410 0x7f25dc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25e0f3e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc106c20 con 0x7f25dc0f5410 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25da59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25dc0f5410 0x7f25dc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25da59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25dc0f5410 0x7f25dc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:48620/0 (socket says 172.21.15.148:48620) 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25da59c700 1 -- 172.21.15.148:0/225537165 learned_addr learned my addr 172.21.15.148:0/225537165 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25da59c700 1 -- 172.21.15.148:0/225537165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc005040 con 0x7f25dc0f5410 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25da59c700 1 --2- 172.21.15.148:0/225537165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25dc0f5410 0x7f25dc106730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f25cc013510 tx=0x7f25cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25caffd700 1 -- 172.21.15.148:0/225537165 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25cc009320 con 0x7f25dc0f5410 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25e0f3e700 1 -- 172.21.15.148:0/225537165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc106e20 con 0x7f25dc0f5410 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25caffd700 1 -- 172.21.15.148:0/225537165 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25cc013f00 con 0x7f25dc0f5410 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25caffd700 1 -- 172.21.15.148:0/225537165 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25cc014d90 con 0x7f25dc0f5410 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.424+0000 7f25e0f3e700 1 -- 172.21.15.148:0/225537165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc1072b0 con 0x7f25dc0f5410 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.425+0000 7f25caffd700 1 -- 172.21.15.148:0/225537165 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25cc0263a0 con 0x7f25dc0f5410 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.425+0000 7f25caffd700 1 --2- 172.21.15.148:0/225537165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f25c402ff60 0x7f25c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.425+0000 7f25caffd700 1 -- 172.21.15.148:0/225537165 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25cc0410a0 con 0x7f25dc0f5410 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.425+0000 7f25e0f3e700 1 -- 172.21.15.148:0/225537165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc059100 con 0x7f25dc0f5410 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.426+0000 7f25d9d9b700 1 --2- 172.21.15.148:0/225537165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f25c402ff60 0x7f25c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.426+0000 7f25d9d9b700 1 --2- 172.21.15.148:0/225537165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f25c402ff60 0x7f25c4032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f25d00096a0 tx=0x7f25d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:03.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.430+0000 7f25caffd700 1 -- 172.21.15.148:0/225537165 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25cc012920 con 0x7f25dc0f5410 2023-12-21T03:41:03.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.636+0000 7f25e0f3e700 1 -- 172.21.15.148:0/225537165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25dc0fae60 con 0x7f25dc0f5410 2023-12-21T03:41:03.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.637+0000 7f25caffd700 1 -- 172.21.15.148:0/225537165 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f25cc02b020 con 0x7f25dc0f5410 2023-12-21T03:41:03.639 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:03.639 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.638+0000 7f25e0f3e700 1 -- 172.21.15.148:0/225537165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f25c402ff60 msgr2=0x7f25c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.638+0000 7f25e0f3e700 1 --2- 172.21.15.148:0/225537165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f25c402ff60 0x7f25c4032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f25d00096a0 tx=0x7f25d0006b40).stop 2023-12-21T03:41:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.638+0000 7f25e0f3e700 1 -- 172.21.15.148:0/225537165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25dc0f5410 msgr2=0x7f25dc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.638+0000 7f25e0f3e700 1 --2- 172.21.15.148:0/225537165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25dc0f5410 0x7f25dc106730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f25cc013510 tx=0x7f25cc00b040).stop 2023-12-21T03:41:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.638+0000 7f25e0f3e700 1 -- 172.21.15.148:0/225537165 shutdown_connections 2023-12-21T03:41:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.638+0000 7f25e0f3e700 1 --2- 172.21.15.148:0/225537165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f25c402ff60 0x7f25c4032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.638+0000 7f25e0f3e700 1 --2- 172.21.15.148:0/225537165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25dc0f5410 0x7f25dc106730 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.638+0000 7f25e0f3e700 1 -- 172.21.15.148:0/225537165 >> 172.21.15.148:0/225537165 conn(0x7f25dc0f0b70 msgr2=0x7f25dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.639+0000 7f25e0f3e700 1 -- 172.21.15.148:0/225537165 shutdown_connections 2023-12-21T03:41:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:03.639+0000 7f25e0f3e700 1 -- 172.21.15.148:0/225537165 wait complete. 2023-12-21T03:41:03.642 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:04.070 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:03.638862+0000 mon.smithi078 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.148:0/225537165' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:05.391 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:05.391 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:05.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:04.415270+0000 mgr.smithi078.zidbgq (mgr.14162) 52 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:06.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.556+0000 7fee73228700 1 -- 172.21.15.148:0/3579962090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee6c060280 msgr2=0x7fee6c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:06.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.556+0000 7fee73228700 1 --2- 172.21.15.148:0/3579962090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee6c060280 0x7fee6c0f90f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fee60009fb0 tx=0x7fee60009b30).stop 2023-12-21T03:41:06.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.556+0000 7fee73228700 1 -- 172.21.15.148:0/3579962090 shutdown_connections 2023-12-21T03:41:06.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.556+0000 7fee73228700 1 --2- 172.21.15.148:0/3579962090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee6c060280 0x7fee6c0f90f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:06.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.556+0000 7fee73228700 1 -- 172.21.15.148:0/3579962090 >> 172.21.15.148:0/3579962090 conn(0x7fee6c0f0b70 msgr2=0x7fee6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:06.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.556+0000 7fee73228700 1 -- 172.21.15.148:0/3579962090 shutdown_connections 2023-12-21T03:41:06.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.556+0000 7fee73228700 1 -- 172.21.15.148:0/3579962090 wait complete. 2023-12-21T03:41:06.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.557+0000 7fee73228700 1 Processor -- start 2023-12-21T03:41:06.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.557+0000 7fee73228700 1 -- start start 2023-12-21T03:41:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.557+0000 7fee73228700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee6c060280 0x7fee6c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.557+0000 7fee73228700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c106bd0 con 0x7fee6c060280 2023-12-21T03:41:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.557+0000 7fee70fc4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee6c060280 0x7fee6c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.558+0000 7fee70fc4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee6c060280 0x7fee6c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:48640/0 (socket says 172.21.15.148:48640) 2023-12-21T03:41:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.558+0000 7fee70fc4700 1 -- 172.21.15.148:0/722389172 learned_addr learned my addr 172.21.15.148:0/722389172 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.558+0000 7fee70fc4700 1 -- 172.21.15.148:0/722389172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee60005040 con 0x7fee6c060280 2023-12-21T03:41:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.558+0000 7fee70fc4700 1 --2- 172.21.15.148:0/722389172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee6c060280 0x7fee6c1066e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fee60013510 tx=0x7fee6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.558+0000 7fee697fa700 1 -- 172.21.15.148:0/722389172 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee60009320 con 0x7fee6c060280 2023-12-21T03:41:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.558+0000 7fee73228700 1 -- 172.21.15.148:0/722389172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6c106dd0 con 0x7fee6c060280 2023-12-21T03:41:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.558+0000 7fee697fa700 1 -- 172.21.15.148:0/722389172 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee60013f00 con 0x7fee6c060280 2023-12-21T03:41:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.558+0000 7fee697fa700 1 -- 172.21.15.148:0/722389172 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee60014dc0 con 0x7fee6c060280 2023-12-21T03:41:06.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.558+0000 7fee73228700 1 -- 172.21.15.148:0/722389172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6c107260 con 0x7fee6c060280 2023-12-21T03:41:06.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.559+0000 7fee697fa700 1 -- 172.21.15.148:0/722389172 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee6001d3d0 con 0x7fee6c060280 2023-12-21T03:41:06.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.559+0000 7fee697fa700 1 --2- 172.21.15.148:0/722389172 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fee5c02ff60 0x7fee5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:06.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.559+0000 7fee697fa700 1 -- 172.21.15.148:0/722389172 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee600145f0 con 0x7fee6c060280 2023-12-21T03:41:06.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.559+0000 7fee73228700 1 -- 172.21.15.148:0/722389172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6c045ad0 con 0x7fee6c060280 2023-12-21T03:41:06.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.560+0000 7fee6bfff700 1 --2- 172.21.15.148:0/722389172 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fee5c02ff60 0x7fee5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:06.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.560+0000 7fee6bfff700 1 --2- 172.21.15.148:0/722389172 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fee5c02ff60 0x7fee5c032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fee580096a0 tx=0x7fee58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:06.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.564+0000 7fee697fa700 1 -- 172.21.15.148:0/722389172 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee60012920 con 0x7fee6c060280 2023-12-21T03:41:06.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.770+0000 7fee73228700 1 -- 172.21.15.148:0/722389172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee6c107ee0 con 0x7fee6c060280 2023-12-21T03:41:06.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.770+0000 7fee697fa700 1 -- 172.21.15.148:0/722389172 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee60012330 con 0x7fee6c060280 2023-12-21T03:41:06.774 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:06.774 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:06.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.772+0000 7fee73228700 1 -- 172.21.15.148:0/722389172 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fee5c02ff60 msgr2=0x7fee5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.772+0000 7fee73228700 1 --2- 172.21.15.148:0/722389172 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fee5c02ff60 0x7fee5c032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fee580096a0 tx=0x7fee58006b40).stop 2023-12-21T03:41:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.772+0000 7fee73228700 1 -- 172.21.15.148:0/722389172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee6c060280 msgr2=0x7fee6c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.772+0000 7fee73228700 1 --2- 172.21.15.148:0/722389172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee6c060280 0x7fee6c1066e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fee60013510 tx=0x7fee6000b040).stop 2023-12-21T03:41:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.772+0000 7fee73228700 1 -- 172.21.15.148:0/722389172 shutdown_connections 2023-12-21T03:41:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.772+0000 7fee73228700 1 --2- 172.21.15.148:0/722389172 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fee5c02ff60 0x7fee5c032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.772+0000 7fee73228700 1 --2- 172.21.15.148:0/722389172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee6c060280 0x7fee6c1066e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.772+0000 7fee73228700 1 -- 172.21.15.148:0/722389172 >> 172.21.15.148:0/722389172 conn(0x7fee6c0f0b70 msgr2=0x7fee6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.772+0000 7fee73228700 1 -- 172.21.15.148:0/722389172 shutdown_connections 2023-12-21T03:41:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:06.772+0000 7fee73228700 1 -- 172.21.15.148:0/722389172 wait complete. 2023-12-21T03:41:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:06.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:06 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:06.415642+0000 mgr.smithi078.zidbgq (mgr.14162) 53 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:06.772360+0000 mon.smithi078 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.148:0/722389172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:08.181 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:08.182 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:08.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:08.415954+0000 mgr.smithi078.zidbgq (mgr.14162) 54 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:09.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.363+0000 7f4afebf3700 1 -- 172.21.15.148:0/2628157427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4af80f6510 msgr2=0x7f4af80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:09.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.363+0000 7f4afebf3700 1 --2- 172.21.15.148:0/2628157427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4af80f6510 0x7f4af80f6930 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f4aec009fb0 tx=0x7f4aec009b30).stop 2023-12-21T03:41:09.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.363+0000 7f4afebf3700 1 -- 172.21.15.148:0/2628157427 shutdown_connections 2023-12-21T03:41:09.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.363+0000 7f4afebf3700 1 --2- 172.21.15.148:0/2628157427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4af80f6510 0x7f4af80f6930 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:09.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.363+0000 7f4afebf3700 1 -- 172.21.15.148:0/2628157427 >> 172.21.15.148:0/2628157427 conn(0x7f4af80f1c70 msgr2=0x7f4af80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:09.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.363+0000 7f4afebf3700 1 -- 172.21.15.148:0/2628157427 shutdown_connections 2023-12-21T03:41:09.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.363+0000 7f4afebf3700 1 -- 172.21.15.148:0/2628157427 wait complete. 2023-12-21T03:41:09.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.364+0000 7f4afebf3700 1 Processor -- start 2023-12-21T03:41:09.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.364+0000 7f4afebf3700 1 -- start start 2023-12-21T03:41:09.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.364+0000 7f4afebf3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4af80f6510 0x7f4af806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:09.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.364+0000 7f4afebf3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8069e20 con 0x7f4af80f6510 2023-12-21T03:41:09.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.365+0000 7f4afc98f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4af80f6510 0x7f4af806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:09.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.365+0000 7f4afc98f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4af80f6510 0x7f4af806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:54868/0 (socket says 172.21.15.148:54868) 2023-12-21T03:41:09.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.365+0000 7f4afc98f700 1 -- 172.21.15.148:0/3305820070 learned_addr learned my addr 172.21.15.148:0/3305820070 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:09.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.365+0000 7f4afc98f700 1 -- 172.21.15.148:0/3305820070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec005040 con 0x7f4af80f6510 2023-12-21T03:41:09.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.365+0000 7f4afc98f700 1 --2- 172.21.15.148:0/3305820070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4af80f6510 0x7f4af806b9a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f4aec013510 tx=0x7f4aec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:09.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.365+0000 7f4af57fa700 1 -- 172.21.15.148:0/3305820070 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aec009320 con 0x7f4af80f6510 2023-12-21T03:41:09.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.365+0000 7f4af57fa700 1 -- 172.21.15.148:0/3305820070 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4aec013f00 con 0x7f4af80f6510 2023-12-21T03:41:09.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.365+0000 7f4afebf3700 1 -- 172.21.15.148:0/3305820070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af8069fc0 con 0x7f4af80f6510 2023-12-21T03:41:09.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.365+0000 7f4af57fa700 1 -- 172.21.15.148:0/3305820070 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aec014d90 con 0x7f4af80f6510 2023-12-21T03:41:09.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.365+0000 7f4afebf3700 1 -- 172.21.15.148:0/3305820070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af806a450 con 0x7f4af80f6510 2023-12-21T03:41:09.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.367+0000 7f4af57fa700 1 -- 172.21.15.148:0/3305820070 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4aec0095f0 con 0x7f4af80f6510 2023-12-21T03:41:09.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.367+0000 7f4afebf3700 1 -- 172.21.15.148:0/3305820070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af8059100 con 0x7f4af80f6510 2023-12-21T03:41:09.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.367+0000 7f4af57fa700 1 --2- 172.21.15.148:0/3305820070 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4ae802ffb0 0x7f4ae8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:09.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.367+0000 7f4af57fa700 1 -- 172.21.15.148:0/3305820070 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4aec040fa0 con 0x7f4af80f6510 2023-12-21T03:41:09.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.367+0000 7f4af7fff700 1 --2- 172.21.15.148:0/3305820070 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4ae802ffb0 0x7f4ae8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:09.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.367+0000 7f4af7fff700 1 --2- 172.21.15.148:0/3305820070 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4ae802ffb0 0x7f4ae8032470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f4ae40096a0 tx=0x7f4ae4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:09.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.372+0000 7f4af57fa700 1 -- 172.21.15.148:0/3305820070 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4aec06d2b0 con 0x7f4af80f6510 2023-12-21T03:41:09.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.577+0000 7f4afebf3700 1 -- 172.21.15.148:0/3305820070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4af8045ad0 con 0x7f4af80f6510 2023-12-21T03:41:09.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.578+0000 7f4af57fa700 1 -- 172.21.15.148:0/3305820070 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4aec014420 con 0x7f4af80f6510 2023-12-21T03:41:09.579 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:09.579 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.580+0000 7f4afebf3700 1 -- 172.21.15.148:0/3305820070 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4ae802ffb0 msgr2=0x7f4ae8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.580+0000 7f4afebf3700 1 --2- 172.21.15.148:0/3305820070 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4ae802ffb0 0x7f4ae8032470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f4ae40096a0 tx=0x7f4ae4006b40).stop 2023-12-21T03:41:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.580+0000 7f4afebf3700 1 -- 172.21.15.148:0/3305820070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4af80f6510 msgr2=0x7f4af806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.580+0000 7f4afebf3700 1 --2- 172.21.15.148:0/3305820070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4af80f6510 0x7f4af806b9a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f4aec013510 tx=0x7f4aec00b040).stop 2023-12-21T03:41:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.580+0000 7f4afebf3700 1 -- 172.21.15.148:0/3305820070 shutdown_connections 2023-12-21T03:41:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.580+0000 7f4afebf3700 1 --2- 172.21.15.148:0/3305820070 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4ae802ffb0 0x7f4ae8032470 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.580+0000 7f4afebf3700 1 --2- 172.21.15.148:0/3305820070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4af80f6510 0x7f4af806b9a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.580+0000 7f4afebf3700 1 -- 172.21.15.148:0/3305820070 >> 172.21.15.148:0/3305820070 conn(0x7f4af80f1c70 msgr2=0x7f4af80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:09.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.580+0000 7f4afebf3700 1 -- 172.21.15.148:0/3305820070 shutdown_connections 2023-12-21T03:41:09.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:09.580+0000 7f4afebf3700 1 -- 172.21.15.148:0/3305820070 wait complete. 2023-12-21T03:41:09.585 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:10.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:10 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:09.578892+0000 mon.smithi078 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.148:0/3305820070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:11.027 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:11.027 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:11.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:10.416370+0000 mgr.smithi078.zidbgq (mgr.14162) 55 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.224+0000 7f7b0bfff700 1 -- 172.21.15.148:0/1143442755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0f51a0 msgr2=0x7f7b0c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.224+0000 7f7b0bfff700 1 --2- 172.21.15.148:0/1143442755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0f51a0 0x7f7b0c0f55c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f7afc009fb0 tx=0x7f7afc009b30).stop 2023-12-21T03:41:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.224+0000 7f7b0bfff700 1 -- 172.21.15.148:0/1143442755 shutdown_connections 2023-12-21T03:41:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.225+0000 7f7b0bfff700 1 --2- 172.21.15.148:0/1143442755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0f51a0 0x7f7b0c0f55c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.225+0000 7f7b0bfff700 1 -- 172.21.15.148:0/1143442755 >> 172.21.15.148:0/1143442755 conn(0x7f7b0c0f09a0 msgr2=0x7f7b0c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.225+0000 7f7b0bfff700 1 -- 172.21.15.148:0/1143442755 shutdown_connections 2023-12-21T03:41:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.225+0000 7f7b0bfff700 1 -- 172.21.15.148:0/1143442755 wait complete. 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.225+0000 7f7b0bfff700 1 Processor -- start 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.225+0000 7f7b0bfff700 1 -- start start 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.226+0000 7f7b0bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0f51a0 0x7f7b0c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.226+0000 7f7b0bfff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c060240 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.226+0000 7f7b0affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0f51a0 0x7f7b0c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.226+0000 7f7b0affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0f51a0 0x7f7b0c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:54890/0 (socket says 172.21.15.148:54890) 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.226+0000 7f7b0affd700 1 -- 172.21.15.148:0/4134775134 learned_addr learned my addr 172.21.15.148:0/4134775134 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.226+0000 7f7b0affd700 1 -- 172.21.15.148:0/4134775134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc005040 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.226+0000 7f7b0affd700 1 --2- 172.21.15.148:0/4134775134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0f51a0 0x7f7b0c1091b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f7afc013510 tx=0x7f7afc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.227+0000 7f7af37fe700 1 -- 172.21.15.148:0/4134775134 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7afc009320 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.227+0000 7f7af37fe700 1 -- 172.21.15.148:0/4134775134 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7afc013f00 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.227+0000 7f7b0bfff700 1 -- 172.21.15.148:0/4134775134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0c060440 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.227+0000 7f7af37fe700 1 -- 172.21.15.148:0/4134775134 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7afc014dc0 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.227+0000 7f7b0bfff700 1 -- 172.21.15.148:0/4134775134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0c109860 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.227+0000 7f7af37fe700 1 -- 172.21.15.148:0/4134775134 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7afc01d3d0 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.228+0000 7f7b0bfff700 1 -- 172.21.15.148:0/4134775134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0c045ad0 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.228+0000 7f7af37fe700 1 --2- 172.21.15.148:0/4134775134 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7af402ff60 0x7f7af4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:12.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.228+0000 7f7af37fe700 1 -- 172.21.15.148:0/4134775134 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7afc041100 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.228+0000 7f7b0a7fc700 1 --2- 172.21.15.148:0/4134775134 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7af402ff60 0x7f7af4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:12.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.228+0000 7f7b0a7fc700 1 --2- 172.21.15.148:0/4134775134 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7af402ff60 0x7f7af4032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f7b000096a0 tx=0x7f7b00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:12.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.232+0000 7f7af37fe700 1 -- 172.21.15.148:0/4134775134 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7afc0126f0 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.438+0000 7f7b0bfff700 1 -- 172.21.15.148:0/4134775134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b0c10a080 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.439+0000 7f7af37fe700 1 -- 172.21.15.148:0/4134775134 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7afc012330 con 0x7f7b0c0f51a0 2023-12-21T03:41:12.440 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:12.440 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:12.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.440+0000 7f7b0bfff700 1 -- 172.21.15.148:0/4134775134 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7af402ff60 msgr2=0x7f7af4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:12.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.440+0000 7f7b0bfff700 1 --2- 172.21.15.148:0/4134775134 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7af402ff60 0x7f7af4032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f7b000096a0 tx=0x7f7b00006b40).stop 2023-12-21T03:41:12.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.440+0000 7f7b0bfff700 1 -- 172.21.15.148:0/4134775134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0f51a0 msgr2=0x7f7b0c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:12.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.440+0000 7f7b0bfff700 1 --2- 172.21.15.148:0/4134775134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0f51a0 0x7f7b0c1091b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f7afc013510 tx=0x7f7afc00b040).stop 2023-12-21T03:41:12.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.440+0000 7f7b0bfff700 1 -- 172.21.15.148:0/4134775134 shutdown_connections 2023-12-21T03:41:12.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.440+0000 7f7b0bfff700 1 --2- 172.21.15.148:0/4134775134 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7af402ff60 0x7f7af4032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:12.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.440+0000 7f7b0bfff700 1 --2- 172.21.15.148:0/4134775134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0f51a0 0x7f7b0c1091b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:12.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.440+0000 7f7b0bfff700 1 -- 172.21.15.148:0/4134775134 >> 172.21.15.148:0/4134775134 conn(0x7f7b0c0f09a0 msgr2=0x7f7b0c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:12.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.440+0000 7f7b0bfff700 1 -- 172.21.15.148:0/4134775134 shutdown_connections 2023-12-21T03:41:12.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:12.440+0000 7f7b0bfff700 1 -- 172.21.15.148:0/4134775134 wait complete. 2023-12-21T03:41:12.442 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:12.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:12.416619+0000 mgr.smithi078.zidbgq (mgr.14162) 56 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:12.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:12.439663+0000 mon.smithi078 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.148:0/4134775134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:13.848 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:13.848 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:15.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.158+0000 7fdd7b587700 1 -- 172.21.15.148:0/2549698089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd740f6510 msgr2=0x7fdd740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.158+0000 7fdd7b587700 1 --2- 172.21.15.148:0/2549698089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd740f6510 0x7fdd740f6930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fdd64009fb0 tx=0x7fdd64009b30).stop 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.158+0000 7fdd7b587700 1 -- 172.21.15.148:0/2549698089 shutdown_connections 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.158+0000 7fdd7b587700 1 --2- 172.21.15.148:0/2549698089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd740f6510 0x7fdd740f6930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.158+0000 7fdd7b587700 1 -- 172.21.15.148:0/2549698089 >> 172.21.15.148:0/2549698089 conn(0x7fdd740f1c70 msgr2=0x7fdd740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.158+0000 7fdd7b587700 1 -- 172.21.15.148:0/2549698089 shutdown_connections 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.158+0000 7fdd7b587700 1 -- 172.21.15.148:0/2549698089 wait complete. 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.159+0000 7fdd7b587700 1 Processor -- start 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.159+0000 7fdd7b587700 1 -- start start 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.159+0000 7fdd7b587700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd740f6510 0x7fdd7406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.159+0000 7fdd7b587700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd74069e20 con 0x7fdd740f6510 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.159+0000 7fdd79323700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd740f6510 0x7fdd7406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.159+0000 7fdd79323700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd740f6510 0x7fdd7406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:54908/0 (socket says 172.21.15.148:54908) 2023-12-21T03:41:15.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.159+0000 7fdd79323700 1 -- 172.21.15.148:0/3757383964 learned_addr learned my addr 172.21.15.148:0/3757383964 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:15.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.159+0000 7fdd79323700 1 -- 172.21.15.148:0/3757383964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd64005040 con 0x7fdd740f6510 2023-12-21T03:41:15.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.160+0000 7fdd79323700 1 --2- 172.21.15.148:0/3757383964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd740f6510 0x7fdd7406b9a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fdd64013510 tx=0x7fdd6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.160+0000 7fdd69ffb700 1 -- 172.21.15.148:0/3757383964 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd64009320 con 0x7fdd740f6510 2023-12-21T03:41:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.160+0000 7fdd7b587700 1 -- 172.21.15.148:0/3757383964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd74069fc0 con 0x7fdd740f6510 2023-12-21T03:41:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.160+0000 7fdd69ffb700 1 -- 172.21.15.148:0/3757383964 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd64013f00 con 0x7fdd740f6510 2023-12-21T03:41:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.160+0000 7fdd69ffb700 1 -- 172.21.15.148:0/3757383964 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd64014d90 con 0x7fdd740f6510 2023-12-21T03:41:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.160+0000 7fdd7b587700 1 -- 172.21.15.148:0/3757383964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7406a450 con 0x7fdd740f6510 2023-12-21T03:41:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.161+0000 7fdd69ffb700 1 -- 172.21.15.148:0/3757383964 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd640263a0 con 0x7fdd740f6510 2023-12-21T03:41:15.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.161+0000 7fdd69ffb700 1 --2- 172.21.15.148:0/3757383964 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdd6002ff60 0x7fdd60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:15.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.161+0000 7fdd69ffb700 1 -- 172.21.15.148:0/3757383964 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd64040ff0 con 0x7fdd740f6510 2023-12-21T03:41:15.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.161+0000 7fdd7b587700 1 -- 172.21.15.148:0/3757383964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd74059100 con 0x7fdd740f6510 2023-12-21T03:41:15.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.161+0000 7fdd78b22700 1 --2- 172.21.15.148:0/3757383964 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdd6002ff60 0x7fdd60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:15.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.161+0000 7fdd78b22700 1 --2- 172.21.15.148:0/3757383964 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdd6002ff60 0x7fdd60032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fdd700096a0 tx=0x7fdd70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:15.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.166+0000 7fdd69ffb700 1 -- 172.21.15.148:0/3757383964 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd64012920 con 0x7fdd740f6510 2023-12-21T03:41:15.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.368+0000 7fdd7b587700 1 -- 172.21.15.148:0/3757383964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd740fbf60 con 0x7fdd740f6510 2023-12-21T03:41:15.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.369+0000 7fdd69ffb700 1 -- 172.21.15.148:0/3757383964 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd6402b020 con 0x7fdd740f6510 2023-12-21T03:41:15.371 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:15.371 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:15.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.371+0000 7fdd7b587700 1 -- 172.21.15.148:0/3757383964 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdd6002ff60 msgr2=0x7fdd60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:15.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.371+0000 7fdd7b587700 1 --2- 172.21.15.148:0/3757383964 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdd6002ff60 0x7fdd60032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fdd700096a0 tx=0x7fdd70006b40).stop 2023-12-21T03:41:15.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.371+0000 7fdd7b587700 1 -- 172.21.15.148:0/3757383964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd740f6510 msgr2=0x7fdd7406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:15.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.371+0000 7fdd7b587700 1 --2- 172.21.15.148:0/3757383964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd740f6510 0x7fdd7406b9a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fdd64013510 tx=0x7fdd6400b040).stop 2023-12-21T03:41:15.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.371+0000 7fdd7b587700 1 -- 172.21.15.148:0/3757383964 shutdown_connections 2023-12-21T03:41:15.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.371+0000 7fdd7b587700 1 --2- 172.21.15.148:0/3757383964 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdd6002ff60 0x7fdd60032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:15.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.371+0000 7fdd7b587700 1 --2- 172.21.15.148:0/3757383964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd740f6510 0x7fdd7406b9a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:15.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.371+0000 7fdd7b587700 1 -- 172.21.15.148:0/3757383964 >> 172.21.15.148:0/3757383964 conn(0x7fdd740f1c70 msgr2=0x7fdd740f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:15.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.371+0000 7fdd7b587700 1 -- 172.21.15.148:0/3757383964 shutdown_connections 2023-12-21T03:41:15.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:15.371+0000 7fdd7b587700 1 -- 172.21.15.148:0/3757383964 wait complete. 2023-12-21T03:41:15.374 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:15.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:14.416890+0000 mgr.smithi078.zidbgq (mgr.14162) 57 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:16.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:16 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:15.370007+0000 mon.smithi078 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.148:0/3757383964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:16.754 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:16.755 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:17.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:16.417256+0000 mgr.smithi078.zidbgq (mgr.14162) 58 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:17.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.941+0000 7fdf71497700 1 -- 172.21.15.148:0/3629185523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf6c0f5410 msgr2=0x7fdf6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:17.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.941+0000 7fdf71497700 1 --2- 172.21.15.148:0/3629185523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c0f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fdf60009fb0 tx=0x7fdf60009b30).stop 2023-12-21T03:41:17.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.941+0000 7fdf71497700 1 -- 172.21.15.148:0/3629185523 shutdown_connections 2023-12-21T03:41:17.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.941+0000 7fdf71497700 1 --2- 172.21.15.148:0/3629185523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c0f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:17.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.941+0000 7fdf71497700 1 -- 172.21.15.148:0/3629185523 >> 172.21.15.148:0/3629185523 conn(0x7fdf6c0f0b70 msgr2=0x7fdf6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.941+0000 7fdf71497700 1 -- 172.21.15.148:0/3629185523 shutdown_connections 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.941+0000 7fdf71497700 1 -- 172.21.15.148:0/3629185523 wait complete. 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.942+0000 7fdf71497700 1 Processor -- start 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.942+0000 7fdf71497700 1 -- start start 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.942+0000 7fdf71497700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.942+0000 7fdf71497700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c060220 con 0x7fdf6c0f5410 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.942+0000 7fdf6affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.942+0000 7fdf6affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:54924/0 (socket says 172.21.15.148:54924) 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.942+0000 7fdf6affd700 1 -- 172.21.15.148:0/2036555598 learned_addr learned my addr 172.21.15.148:0/2036555598 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.942+0000 7fdf6affd700 1 -- 172.21.15.148:0/2036555598 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf60005040 con 0x7fdf6c0f5410 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.942+0000 7fdf6affd700 1 --2- 172.21.15.148:0/2036555598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fdf60013510 tx=0x7fdf6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:17.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.943+0000 7fdf537fe700 1 -- 172.21.15.148:0/2036555598 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf60009320 con 0x7fdf6c0f5410 2023-12-21T03:41:17.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.943+0000 7fdf537fe700 1 -- 172.21.15.148:0/2036555598 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf60013f00 con 0x7fdf6c0f5410 2023-12-21T03:41:17.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.943+0000 7fdf71497700 1 -- 172.21.15.148:0/2036555598 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c060420 con 0x7fdf6c0f5410 2023-12-21T03:41:17.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.943+0000 7fdf537fe700 1 -- 172.21.15.148:0/2036555598 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf60014d90 con 0x7fdf6c0f5410 2023-12-21T03:41:17.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.943+0000 7fdf71497700 1 -- 172.21.15.148:0/2036555598 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c109ad0 con 0x7fdf6c0f5410 2023-12-21T03:41:17.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.943+0000 7fdf537fe700 1 -- 172.21.15.148:0/2036555598 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf600263a0 con 0x7fdf6c0f5410 2023-12-21T03:41:17.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.943+0000 7fdf71497700 1 -- 172.21.15.148:0/2036555598 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c059100 con 0x7fdf6c0f5410 2023-12-21T03:41:17.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.944+0000 7fdf537fe700 1 --2- 172.21.15.148:0/2036555598 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdf5402ffb0 0x7fdf54032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:17.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.944+0000 7fdf537fe700 1 -- 172.21.15.148:0/2036555598 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf60041370 con 0x7fdf6c0f5410 2023-12-21T03:41:17.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.944+0000 7fdf6a7fc700 1 --2- 172.21.15.148:0/2036555598 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdf5402ffb0 0x7fdf54032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:17.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.944+0000 7fdf6a7fc700 1 --2- 172.21.15.148:0/2036555598 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdf5402ffb0 0x7fdf54032470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fdf5c0096a0 tx=0x7fdf5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:17.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:17.949+0000 7fdf537fe700 1 -- 172.21.15.148:0/2036555598 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf600126f0 con 0x7fdf6c0f5410 2023-12-21T03:41:18.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.153+0000 7fdf71497700 1 -- 172.21.15.148:0/2036555598 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf6c045ad0 con 0x7fdf6c0f5410 2023-12-21T03:41:18.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.153+0000 7fdf537fe700 1 -- 172.21.15.148:0/2036555598 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf60012330 con 0x7fdf6c0f5410 2023-12-21T03:41:18.155 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:18.155 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:18.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.155+0000 7fdf71497700 1 -- 172.21.15.148:0/2036555598 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdf5402ffb0 msgr2=0x7fdf54032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:18.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.155+0000 7fdf71497700 1 --2- 172.21.15.148:0/2036555598 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdf5402ffb0 0x7fdf54032470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fdf5c0096a0 tx=0x7fdf5c006b40).stop 2023-12-21T03:41:18.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.155+0000 7fdf71497700 1 -- 172.21.15.148:0/2036555598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf6c0f5410 msgr2=0x7fdf6c109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:18.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.155+0000 7fdf71497700 1 --2- 172.21.15.148:0/2036555598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fdf60013510 tx=0x7fdf6000b040).stop 2023-12-21T03:41:18.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.155+0000 7fdf71497700 1 -- 172.21.15.148:0/2036555598 shutdown_connections 2023-12-21T03:41:18.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.155+0000 7fdf71497700 1 --2- 172.21.15.148:0/2036555598 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdf5402ffb0 0x7fdf54032470 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:18.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.155+0000 7fdf71497700 1 --2- 172.21.15.148:0/2036555598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:18.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.155+0000 7fdf71497700 1 -- 172.21.15.148:0/2036555598 >> 172.21.15.148:0/2036555598 conn(0x7fdf6c0f0b70 msgr2=0x7fdf6c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:18.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.155+0000 7fdf71497700 1 -- 172.21.15.148:0/2036555598 shutdown_connections 2023-12-21T03:41:18.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:18.155+0000 7fdf71497700 1 -- 172.21.15.148:0/2036555598 wait complete. 2023-12-21T03:41:18.157 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:18.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:18 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:18.154214+0000 mon.smithi078 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.148:0/2036555598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:19.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:19.530 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:19.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:18.417575+0000 mgr.smithi078.zidbgq (mgr.14162) 59 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:20.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.616+0000 7f55fcb18700 1 -- 172.21.15.148:0/2662289404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f80f5410 msgr2=0x7f55f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:20.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.616+0000 7f55fcb18700 1 --2- 172.21.15.148:0/2662289404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f80f5410 0x7f55f80f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f55e8009fb0 tx=0x7f55e8009b30).stop 2023-12-21T03:41:20.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.616+0000 7f55fcb18700 1 -- 172.21.15.148:0/2662289404 shutdown_connections 2023-12-21T03:41:20.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.616+0000 7f55fcb18700 1 --2- 172.21.15.148:0/2662289404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f80f5410 0x7f55f80f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:20.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.616+0000 7f55fcb18700 1 -- 172.21.15.148:0/2662289404 >> 172.21.15.148:0/2662289404 conn(0x7f55f80f0b70 msgr2=0x7f55f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:20.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.616+0000 7f55fcb18700 1 -- 172.21.15.148:0/2662289404 shutdown_connections 2023-12-21T03:41:20.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.616+0000 7f55fcb18700 1 -- 172.21.15.148:0/2662289404 wait complete. 2023-12-21T03:41:20.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.617+0000 7f55fcb18700 1 Processor -- start 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.617+0000 7f55fcb18700 1 -- start start 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.617+0000 7f55fcb18700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f80f5410 0x7f55f8109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.617+0000 7f55fcb18700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f8060240 con 0x7f55f80f5410 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.617+0000 7f55f659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f80f5410 0x7f55f8109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.617+0000 7f55f659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f80f5410 0x7f55f8109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:39028/0 (socket says 172.21.15.148:39028) 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.617+0000 7f55f659c700 1 -- 172.21.15.148:0/3332385991 learned_addr learned my addr 172.21.15.148:0/3332385991 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.617+0000 7f55f659c700 1 -- 172.21.15.148:0/3332385991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e8005040 con 0x7f55f80f5410 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.617+0000 7f55f659c700 1 --2- 172.21.15.148:0/3332385991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f80f5410 0x7f55f8109400 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f55e8013510 tx=0x7f55e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.618+0000 7f55eeffd700 1 -- 172.21.15.148:0/3332385991 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e8009320 con 0x7f55f80f5410 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.618+0000 7f55fcb18700 1 -- 172.21.15.148:0/3332385991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f8060440 con 0x7f55f80f5410 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.618+0000 7f55eeffd700 1 -- 172.21.15.148:0/3332385991 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55e8013f00 con 0x7f55f80f5410 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.618+0000 7f55eeffd700 1 -- 172.21.15.148:0/3332385991 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e8014d90 con 0x7f55f80f5410 2023-12-21T03:41:20.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.618+0000 7f55fcb18700 1 -- 172.21.15.148:0/3332385991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f8109ab0 con 0x7f55f80f5410 2023-12-21T03:41:20.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.618+0000 7f55eeffd700 1 -- 172.21.15.148:0/3332385991 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55e80263a0 con 0x7f55f80f5410 2023-12-21T03:41:20.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.619+0000 7f55eeffd700 1 --2- 172.21.15.148:0/3332385991 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f55e402ff60 0x7f55e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:20.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.619+0000 7f55eeffd700 1 -- 172.21.15.148:0/3332385991 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55e8040ff0 con 0x7f55f80f5410 2023-12-21T03:41:20.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.619+0000 7f55fcb18700 1 -- 172.21.15.148:0/3332385991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f8045ad0 con 0x7f55f80f5410 2023-12-21T03:41:20.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.619+0000 7f55f5d9b700 1 --2- 172.21.15.148:0/3332385991 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f55e402ff60 0x7f55e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:20.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.619+0000 7f55f5d9b700 1 --2- 172.21.15.148:0/3332385991 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f55e402ff60 0x7f55e4032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f55e00096a0 tx=0x7f55e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:20.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.624+0000 7f55eeffd700 1 -- 172.21.15.148:0/3332385991 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55e8012920 con 0x7f55f80f5410 2023-12-21T03:41:20.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:20 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:20.417875+0000 mgr.smithi078.zidbgq (mgr.14162) 60 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:20.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.833+0000 7f55fcb18700 1 -- 172.21.15.148:0/3332385991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55f810a160 con 0x7f55f80f5410 2023-12-21T03:41:20.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.834+0000 7f55eeffd700 1 -- 172.21.15.148:0/3332385991 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f55e802b020 con 0x7f55f80f5410 2023-12-21T03:41:20.835 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:20.835 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:20.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.835+0000 7f55fcb18700 1 -- 172.21.15.148:0/3332385991 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f55e402ff60 msgr2=0x7f55e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:20.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.835+0000 7f55fcb18700 1 --2- 172.21.15.148:0/3332385991 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f55e402ff60 0x7f55e4032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f55e00096a0 tx=0x7f55e0006b40).stop 2023-12-21T03:41:20.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.835+0000 7f55fcb18700 1 -- 172.21.15.148:0/3332385991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f80f5410 msgr2=0x7f55f8109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:20.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.835+0000 7f55fcb18700 1 --2- 172.21.15.148:0/3332385991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f80f5410 0x7f55f8109400 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f55e8013510 tx=0x7f55e800b040).stop 2023-12-21T03:41:20.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.835+0000 7f55fcb18700 1 -- 172.21.15.148:0/3332385991 shutdown_connections 2023-12-21T03:41:20.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.835+0000 7f55fcb18700 1 --2- 172.21.15.148:0/3332385991 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f55e402ff60 0x7f55e4032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:20.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.835+0000 7f55fcb18700 1 --2- 172.21.15.148:0/3332385991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f80f5410 0x7f55f8109400 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:20.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.835+0000 7f55fcb18700 1 -- 172.21.15.148:0/3332385991 >> 172.21.15.148:0/3332385991 conn(0x7f55f80f0b70 msgr2=0x7f55f80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:20.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.835+0000 7f55fcb18700 1 -- 172.21.15.148:0/3332385991 shutdown_connections 2023-12-21T03:41:20.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:20.835+0000 7f55fcb18700 1 -- 172.21.15.148:0/3332385991 wait complete. 2023-12-21T03:41:20.837 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:20.834493+0000 mon.smithi078 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.148:0/3332385991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:22.205 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:22.206 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:22.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:22.418179+0000 mgr.smithi078.zidbgq (mgr.14162) 61 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:23.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.287+0000 7f31b3fff700 1 -- 172.21.15.148:0/2423256927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b40f51a0 msgr2=0x7f31b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:23.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.287+0000 7f31b3fff700 1 --2- 172.21.15.148:0/2423256927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b40f51a0 0x7f31b40f55c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f31a4009fb0 tx=0x7f31a4009b30).stop 2023-12-21T03:41:23.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.287+0000 7f31b3fff700 1 -- 172.21.15.148:0/2423256927 shutdown_connections 2023-12-21T03:41:23.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.287+0000 7f31b3fff700 1 --2- 172.21.15.148:0/2423256927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b40f51a0 0x7f31b40f55c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:23.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.287+0000 7f31b3fff700 1 -- 172.21.15.148:0/2423256927 >> 172.21.15.148:0/2423256927 conn(0x7f31b40f09a0 msgr2=0x7f31b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:23.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.287+0000 7f31b3fff700 1 -- 172.21.15.148:0/2423256927 shutdown_connections 2023-12-21T03:41:23.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.287+0000 7f31b3fff700 1 -- 172.21.15.148:0/2423256927 wait complete. 2023-12-21T03:41:23.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.288+0000 7f31b3fff700 1 Processor -- start 2023-12-21T03:41:23.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.288+0000 7f31b3fff700 1 -- start start 2023-12-21T03:41:23.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.288+0000 7f31b3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b40f51a0 0x7f31b41091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:23.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.288+0000 7f31b3fff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b4060240 con 0x7f31b40f51a0 2023-12-21T03:41:23.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.288+0000 7f31b2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b40f51a0 0x7f31b41091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:23.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.289+0000 7f31b2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b40f51a0 0x7f31b41091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:39048/0 (socket says 172.21.15.148:39048) 2023-12-21T03:41:23.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.289+0000 7f31b2ffd700 1 -- 172.21.15.148:0/3764756341 learned_addr learned my addr 172.21.15.148:0/3764756341 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:23.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.289+0000 7f31b2ffd700 1 -- 172.21.15.148:0/3764756341 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a4005040 con 0x7f31b40f51a0 2023-12-21T03:41:23.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.289+0000 7f31b2ffd700 1 --2- 172.21.15.148:0/3764756341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b40f51a0 0x7f31b41091b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f31a4013510 tx=0x7f31a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:23.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.289+0000 7f31ab7fe700 1 -- 172.21.15.148:0/3764756341 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31a4009320 con 0x7f31b40f51a0 2023-12-21T03:41:23.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.289+0000 7f31b3fff700 1 -- 172.21.15.148:0/3764756341 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b4060440 con 0x7f31b40f51a0 2023-12-21T03:41:23.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.289+0000 7f31ab7fe700 1 -- 172.21.15.148:0/3764756341 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31a4013f00 con 0x7f31b40f51a0 2023-12-21T03:41:23.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.289+0000 7f31ab7fe700 1 -- 172.21.15.148:0/3764756341 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31a4014dc0 con 0x7f31b40f51a0 2023-12-21T03:41:23.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.289+0000 7f31b3fff700 1 -- 172.21.15.148:0/3764756341 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b4109860 con 0x7f31b40f51a0 2023-12-21T03:41:23.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.290+0000 7f31ab7fe700 1 -- 172.21.15.148:0/3764756341 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31a4014420 con 0x7f31b40f51a0 2023-12-21T03:41:23.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.290+0000 7f31b3fff700 1 -- 172.21.15.148:0/3764756341 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b4045ad0 con 0x7f31b40f51a0 2023-12-21T03:41:23.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.290+0000 7f31ab7fe700 1 --2- 172.21.15.148:0/3764756341 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f31a002ff60 0x7f31a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:23.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.290+0000 7f31ab7fe700 1 -- 172.21.15.148:0/3764756341 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f31a4041070 con 0x7f31b40f51a0 2023-12-21T03:41:23.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.290+0000 7f31b27fc700 1 --2- 172.21.15.148:0/3764756341 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f31a002ff60 0x7f31a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:23.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.291+0000 7f31b27fc700 1 --2- 172.21.15.148:0/3764756341 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f31a002ff60 0x7f31a0032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f319c0096a0 tx=0x7f319c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:23.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.295+0000 7f31ab7fe700 1 -- 172.21.15.148:0/3764756341 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31a40095f0 con 0x7f31b40f51a0 2023-12-21T03:41:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.499+0000 7f31b3fff700 1 -- 172.21.15.148:0/3764756341 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31b410a080 con 0x7f31b40f51a0 2023-12-21T03:41:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.500+0000 7f31ab7fe700 1 -- 172.21.15.148:0/3764756341 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f31a401cb10 con 0x7f31b40f51a0 2023-12-21T03:41:23.502 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:23.502 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.502+0000 7f31b3fff700 1 -- 172.21.15.148:0/3764756341 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f31a002ff60 msgr2=0x7f31a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.502+0000 7f31b3fff700 1 --2- 172.21.15.148:0/3764756341 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f31a002ff60 0x7f31a0032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f319c0096a0 tx=0x7f319c006b40).stop 2023-12-21T03:41:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.502+0000 7f31b3fff700 1 -- 172.21.15.148:0/3764756341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b40f51a0 msgr2=0x7f31b41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.502+0000 7f31b3fff700 1 --2- 172.21.15.148:0/3764756341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b40f51a0 0x7f31b41091b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f31a4013510 tx=0x7f31a400b040).stop 2023-12-21T03:41:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.502+0000 7f31b3fff700 1 -- 172.21.15.148:0/3764756341 shutdown_connections 2023-12-21T03:41:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.502+0000 7f31b3fff700 1 --2- 172.21.15.148:0/3764756341 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f31a002ff60 0x7f31a0032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.502+0000 7f31b3fff700 1 --2- 172.21.15.148:0/3764756341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b40f51a0 0x7f31b41091b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.502+0000 7f31b3fff700 1 -- 172.21.15.148:0/3764756341 >> 172.21.15.148:0/3764756341 conn(0x7f31b40f09a0 msgr2=0x7f31b40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.502+0000 7f31b3fff700 1 -- 172.21.15.148:0/3764756341 shutdown_connections 2023-12-21T03:41:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:23.502+0000 7f31b3fff700 1 -- 172.21.15.148:0/3764756341 wait complete. 2023-12-21T03:41:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:23.500999+0000 mon.smithi078 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.148:0/3764756341' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:24.891 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:24.892 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:25.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:24.418496+0000 mgr.smithi078.zidbgq (mgr.14162) 62 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.986+0000 7f20030b6700 1 -- 172.21.15.148:0/1110121700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ffc0f5410 msgr2=0x7f1ffc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.986+0000 7f20030b6700 1 --2- 172.21.15.148:0/1110121700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ffc0f5410 0x7f1ffc0f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f1ff0009fb0 tx=0x7f1ff0009b30).stop 2023-12-21T03:41:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.986+0000 7f20030b6700 1 -- 172.21.15.148:0/1110121700 shutdown_connections 2023-12-21T03:41:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.986+0000 7f20030b6700 1 --2- 172.21.15.148:0/1110121700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ffc0f5410 0x7f1ffc0f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.986+0000 7f20030b6700 1 -- 172.21.15.148:0/1110121700 >> 172.21.15.148:0/1110121700 conn(0x7f1ffc0f0b70 msgr2=0x7f1ffc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.986+0000 7f20030b6700 1 -- 172.21.15.148:0/1110121700 shutdown_connections 2023-12-21T03:41:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.987+0000 7f20030b6700 1 -- 172.21.15.148:0/1110121700 wait complete. 2023-12-21T03:41:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.987+0000 7f20030b6700 1 Processor -- start 2023-12-21T03:41:25.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.987+0000 7f20030b6700 1 -- start start 2023-12-21T03:41:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.988+0000 7f20030b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ffc0f5410 0x7f1ffc105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.988+0000 7f20030b6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc105af0 con 0x7f1ffc0f5410 2023-12-21T03:41:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.988+0000 7f2000e52700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ffc0f5410 0x7f1ffc105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.988+0000 7f2000e52700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ffc0f5410 0x7f1ffc105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:39074/0 (socket says 172.21.15.148:39074) 2023-12-21T03:41:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.988+0000 7f2000e52700 1 -- 172.21.15.148:0/3205204123 learned_addr learned my addr 172.21.15.148:0/3205204123 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.988+0000 7f2000e52700 1 -- 172.21.15.148:0/3205204123 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff0005040 con 0x7f1ffc0f5410 2023-12-21T03:41:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.988+0000 7f2000e52700 1 --2- 172.21.15.148:0/3205204123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ffc0f5410 0x7f1ffc105600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f1ff0013510 tx=0x7f1ff000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.989+0000 7f1ff97fa700 1 -- 172.21.15.148:0/3205204123 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ff0009320 con 0x7f1ffc0f5410 2023-12-21T03:41:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.989+0000 7f20030b6700 1 -- 172.21.15.148:0/3205204123 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc105cf0 con 0x7f1ffc0f5410 2023-12-21T03:41:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.989+0000 7f1ff97fa700 1 -- 172.21.15.148:0/3205204123 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ff0013f00 con 0x7f1ffc0f5410 2023-12-21T03:41:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.989+0000 7f1ff97fa700 1 -- 172.21.15.148:0/3205204123 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ff0014d90 con 0x7f1ffc0f5410 2023-12-21T03:41:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.989+0000 7f20030b6700 1 -- 172.21.15.148:0/3205204123 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc106180 con 0x7f1ffc0f5410 2023-12-21T03:41:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.989+0000 7f1ff97fa700 1 -- 172.21.15.148:0/3205204123 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ff00095f0 con 0x7f1ffc0f5410 2023-12-21T03:41:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.989+0000 7f20030b6700 1 -- 172.21.15.148:0/3205204123 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc045ad0 con 0x7f1ffc0f5410 2023-12-21T03:41:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.990+0000 7f1ff97fa700 1 --2- 172.21.15.148:0/3205204123 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1fec02ff60 0x7f1fec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.990+0000 7f1ff97fa700 1 -- 172.21.15.148:0/3205204123 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ff00410c0 con 0x7f1ffc0f5410 2023-12-21T03:41:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.990+0000 7f1ffbfff700 1 --2- 172.21.15.148:0/3205204123 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1fec02ff60 0x7f1fec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.990+0000 7f1ffbfff700 1 --2- 172.21.15.148:0/3205204123 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1fec02ff60 0x7f1fec032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1fe80096a0 tx=0x7f1fe8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:25.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:25.994+0000 7f1ff97fa700 1 -- 172.21.15.148:0/3205204123 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ff0014420 con 0x7f1ffc0f5410 2023-12-21T03:41:26.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.199+0000 7f20030b6700 1 -- 172.21.15.148:0/3205204123 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ffc0fae60 con 0x7f1ffc0f5410 2023-12-21T03:41:26.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.200+0000 7f1ff97fa700 1 -- 172.21.15.148:0/3205204123 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ff0010230 con 0x7f1ffc0f5410 2023-12-21T03:41:26.201 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:26.201 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.201+0000 7f20030b6700 1 -- 172.21.15.148:0/3205204123 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1fec02ff60 msgr2=0x7f1fec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.201+0000 7f20030b6700 1 --2- 172.21.15.148:0/3205204123 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1fec02ff60 0x7f1fec032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1fe80096a0 tx=0x7f1fe8006b40).stop 2023-12-21T03:41:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.201+0000 7f20030b6700 1 -- 172.21.15.148:0/3205204123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ffc0f5410 msgr2=0x7f1ffc105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.201+0000 7f20030b6700 1 --2- 172.21.15.148:0/3205204123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ffc0f5410 0x7f1ffc105600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f1ff0013510 tx=0x7f1ff000b040).stop 2023-12-21T03:41:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.202+0000 7f20030b6700 1 -- 172.21.15.148:0/3205204123 shutdown_connections 2023-12-21T03:41:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.202+0000 7f20030b6700 1 --2- 172.21.15.148:0/3205204123 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1fec02ff60 0x7f1fec032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.202+0000 7f20030b6700 1 --2- 172.21.15.148:0/3205204123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ffc0f5410 0x7f1ffc105600 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.202+0000 7f20030b6700 1 -- 172.21.15.148:0/3205204123 >> 172.21.15.148:0/3205204123 conn(0x7f1ffc0f0b70 msgr2=0x7f1ffc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.202+0000 7f20030b6700 1 -- 172.21.15.148:0/3205204123 shutdown_connections 2023-12-21T03:41:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:26.202+0000 7f20030b6700 1 -- 172.21.15.148:0/3205204123 wait complete. 2023-12-21T03:41:26.204 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:26.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:26.200420+0000 mon.smithi078 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.148:0/3205204123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:27.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:27.535 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:27.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:26.418855+0000 mgr.smithi078.zidbgq (mgr.14162) 63 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:28.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.623+0000 7f669b59e700 1 -- 172.21.15.148:0/3952158535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c0f51a0 msgr2=0x7f669c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:28.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.623+0000 7f669b59e700 1 --2- 172.21.15.148:0/3952158535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c0f51a0 0x7f669c0f55c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f6690009fb0 tx=0x7f6690009b30).stop 2023-12-21T03:41:28.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.623+0000 7f669b59e700 1 -- 172.21.15.148:0/3952158535 shutdown_connections 2023-12-21T03:41:28.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.623+0000 7f669b59e700 1 --2- 172.21.15.148:0/3952158535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c0f51a0 0x7f669c0f55c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:28.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.623+0000 7f669b59e700 1 -- 172.21.15.148:0/3952158535 >> 172.21.15.148:0/3952158535 conn(0x7f669c0f09a0 msgr2=0x7f669c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:28.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.623+0000 7f669b59e700 1 -- 172.21.15.148:0/3952158535 shutdown_connections 2023-12-21T03:41:28.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.623+0000 7f669b59e700 1 -- 172.21.15.148:0/3952158535 wait complete. 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.623+0000 7f669b59e700 1 Processor -- start 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.623+0000 7f669b59e700 1 -- start start 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.623+0000 7f669b59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c0f51a0 0x7f669c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.623+0000 7f669b59e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c1058a0 con 0x7f669c0f51a0 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.624+0000 7f669a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c0f51a0 0x7f669c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.624+0000 7f669a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c0f51a0 0x7f669c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:39098/0 (socket says 172.21.15.148:39098) 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.624+0000 7f669a59c700 1 -- 172.21.15.148:0/3848744883 learned_addr learned my addr 172.21.15.148:0/3848744883 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.624+0000 7f669a59c700 1 -- 172.21.15.148:0/3848744883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6690005040 con 0x7f669c0f51a0 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.624+0000 7f669a59c700 1 --2- 172.21.15.148:0/3848744883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c0f51a0 0x7f669c1053b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f6690013510 tx=0x7f669000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.624+0000 7f668affd700 1 -- 172.21.15.148:0/3848744883 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6690009320 con 0x7f669c0f51a0 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.624+0000 7f669b59e700 1 -- 172.21.15.148:0/3848744883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669c105aa0 con 0x7f669c0f51a0 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.624+0000 7f668affd700 1 -- 172.21.15.148:0/3848744883 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6690013f00 con 0x7f669c0f51a0 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.624+0000 7f668affd700 1 -- 172.21.15.148:0/3848744883 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6690014dc0 con 0x7f669c0f51a0 2023-12-21T03:41:28.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.624+0000 7f669b59e700 1 -- 172.21.15.148:0/3848744883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669c105f30 con 0x7f669c0f51a0 2023-12-21T03:41:28.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.625+0000 7f668affd700 1 -- 172.21.15.148:0/3848744883 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66900095f0 con 0x7f669c0f51a0 2023-12-21T03:41:28.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.625+0000 7f669b59e700 1 -- 172.21.15.148:0/3848744883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669c045ad0 con 0x7f669c0f51a0 2023-12-21T03:41:28.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.625+0000 7f668affd700 1 --2- 172.21.15.148:0/3848744883 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f668402ff60 0x7f6684032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:28.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.625+0000 7f668affd700 1 -- 172.21.15.148:0/3848744883 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6690041020 con 0x7f669c0f51a0 2023-12-21T03:41:28.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.626+0000 7f6699d9b700 1 --2- 172.21.15.148:0/3848744883 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f668402ff60 0x7f6684032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:28.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.626+0000 7f6699d9b700 1 --2- 172.21.15.148:0/3848744883 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f668402ff60 0x7f6684032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f668c0096a0 tx=0x7f668c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:28.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.630+0000 7f668affd700 1 -- 172.21.15.148:0/3848744883 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6690014420 con 0x7f669c0f51a0 2023-12-21T03:41:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:28.419164+0000 mgr.smithi078.zidbgq (mgr.14162) 64 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:28.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.836+0000 7f669b59e700 1 -- 172.21.15.148:0/3848744883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f669c106d80 con 0x7f669c0f51a0 2023-12-21T03:41:28.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.837+0000 7f668affd700 1 -- 172.21.15.148:0/3848744883 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f669001c920 con 0x7f669c0f51a0 2023-12-21T03:41:28.838 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:28.839 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:28.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.839+0000 7f669b59e700 1 -- 172.21.15.148:0/3848744883 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f668402ff60 msgr2=0x7f6684032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:28.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.839+0000 7f669b59e700 1 --2- 172.21.15.148:0/3848744883 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f668402ff60 0x7f6684032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f668c0096a0 tx=0x7f668c006b40).stop 2023-12-21T03:41:28.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.839+0000 7f669b59e700 1 -- 172.21.15.148:0/3848744883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c0f51a0 msgr2=0x7f669c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:28.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.839+0000 7f669b59e700 1 --2- 172.21.15.148:0/3848744883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c0f51a0 0x7f669c1053b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f6690013510 tx=0x7f669000b040).stop 2023-12-21T03:41:28.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.839+0000 7f669b59e700 1 -- 172.21.15.148:0/3848744883 shutdown_connections 2023-12-21T03:41:28.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.839+0000 7f669b59e700 1 --2- 172.21.15.148:0/3848744883 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f668402ff60 0x7f6684032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:28.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.839+0000 7f669b59e700 1 --2- 172.21.15.148:0/3848744883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c0f51a0 0x7f669c1053b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:28.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.839+0000 7f669b59e700 1 -- 172.21.15.148:0/3848744883 >> 172.21.15.148:0/3848744883 conn(0x7f669c0f09a0 msgr2=0x7f669c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:28.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.839+0000 7f669b59e700 1 -- 172.21.15.148:0/3848744883 shutdown_connections 2023-12-21T03:41:28.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:28.839+0000 7f669b59e700 1 -- 172.21.15.148:0/3848744883 wait complete. 2023-12-21T03:41:28.841 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:28.837748+0000 mon.smithi078 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.148:0/3848744883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:30.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:30.221 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:30.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:30.419497+0000 mgr.smithi078.zidbgq (mgr.14162) 65 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:31.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.361+0000 7fe007844700 1 -- 172.21.15.148:0/1883081780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe000060280 msgr2=0x7fe0000f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.361+0000 7fe007844700 1 --2- 172.21.15.148:0/1883081780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe000060280 0x7fe0000f90f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fdffc009fb0 tx=0x7fdffc009b30).stop 2023-12-21T03:41:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.361+0000 7fe007844700 1 -- 172.21.15.148:0/1883081780 shutdown_connections 2023-12-21T03:41:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.361+0000 7fe007844700 1 --2- 172.21.15.148:0/1883081780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe000060280 0x7fe0000f90f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.361+0000 7fe007844700 1 -- 172.21.15.148:0/1883081780 >> 172.21.15.148:0/1883081780 conn(0x7fe0000f0b70 msgr2=0x7fe0000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.361+0000 7fe007844700 1 -- 172.21.15.148:0/1883081780 shutdown_connections 2023-12-21T03:41:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.361+0000 7fe007844700 1 -- 172.21.15.148:0/1883081780 wait complete. 2023-12-21T03:41:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.361+0000 7fe007844700 1 Processor -- start 2023-12-21T03:41:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.362+0000 7fe007844700 1 -- start start 2023-12-21T03:41:31.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.362+0000 7fe007844700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe000060280 0x7fe0001066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:31.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.362+0000 7fe007844700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe000106bd0 con 0x7fe000060280 2023-12-21T03:41:31.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.362+0000 7fe0055e0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe000060280 0x7fe0001066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:31.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.362+0000 7fe0055e0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe000060280 0x7fe0001066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:47348/0 (socket says 172.21.15.148:47348) 2023-12-21T03:41:31.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.362+0000 7fe0055e0700 1 -- 172.21.15.148:0/1256892192 learned_addr learned my addr 172.21.15.148:0/1256892192 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:31.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.362+0000 7fe0055e0700 1 -- 172.21.15.148:0/1256892192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc005040 con 0x7fe000060280 2023-12-21T03:41:31.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.362+0000 7fe0055e0700 1 --2- 172.21.15.148:0/1256892192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe000060280 0x7fe0001066e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fdffc013510 tx=0x7fdffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:31.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.362+0000 7fdff5ffb700 1 -- 172.21.15.148:0/1256892192 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc009320 con 0x7fe000060280 2023-12-21T03:41:31.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.363+0000 7fdff5ffb700 1 -- 172.21.15.148:0/1256892192 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdffc013f00 con 0x7fe000060280 2023-12-21T03:41:31.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.363+0000 7fe007844700 1 -- 172.21.15.148:0/1256892192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe000106dd0 con 0x7fe000060280 2023-12-21T03:41:31.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.363+0000 7fdff5ffb700 1 -- 172.21.15.148:0/1256892192 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc014dc0 con 0x7fe000060280 2023-12-21T03:41:31.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.363+0000 7fe007844700 1 -- 172.21.15.148:0/1256892192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe000107260 con 0x7fe000060280 2023-12-21T03:41:31.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.364+0000 7fdff5ffb700 1 -- 172.21.15.148:0/1256892192 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdffc0095f0 con 0x7fe000060280 2023-12-21T03:41:31.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.364+0000 7fe007844700 1 -- 172.21.15.148:0/1256892192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe000045ad0 con 0x7fe000060280 2023-12-21T03:41:31.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.364+0000 7fdff5ffb700 1 --2- 172.21.15.148:0/1256892192 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfec02ff60 0x7fdfec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:31.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.364+0000 7fdff5ffb700 1 -- 172.21.15.148:0/1256892192 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdffc041040 con 0x7fe000060280 2023-12-21T03:41:31.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.364+0000 7fe004ddf700 1 --2- 172.21.15.148:0/1256892192 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfec02ff60 0x7fdfec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:31.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.365+0000 7fe004ddf700 1 --2- 172.21.15.148:0/1256892192 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfec02ff60 0x7fdfec032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fdff00096a0 tx=0x7fdff0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:31.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.369+0000 7fdff5ffb700 1 -- 172.21.15.148:0/1256892192 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdffc014420 con 0x7fe000060280 2023-12-21T03:41:31.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.577+0000 7fe007844700 1 -- 172.21.15.148:0/1256892192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe000108110 con 0x7fe000060280 2023-12-21T03:41:31.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.577+0000 7fdff5ffb700 1 -- 172.21.15.148:0/1256892192 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdffc010230 con 0x7fe000060280 2023-12-21T03:41:31.579 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:31.579 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.579+0000 7fe007844700 1 -- 172.21.15.148:0/1256892192 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfec02ff60 msgr2=0x7fdfec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.579+0000 7fe007844700 1 --2- 172.21.15.148:0/1256892192 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfec02ff60 0x7fdfec032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fdff00096a0 tx=0x7fdff0006b40).stop 2023-12-21T03:41:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.579+0000 7fe007844700 1 -- 172.21.15.148:0/1256892192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe000060280 msgr2=0x7fe0001066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.579+0000 7fe007844700 1 --2- 172.21.15.148:0/1256892192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe000060280 0x7fe0001066e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fdffc013510 tx=0x7fdffc00b040).stop 2023-12-21T03:41:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.579+0000 7fe007844700 1 -- 172.21.15.148:0/1256892192 shutdown_connections 2023-12-21T03:41:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.579+0000 7fe007844700 1 --2- 172.21.15.148:0/1256892192 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfec02ff60 0x7fdfec032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.579+0000 7fe007844700 1 --2- 172.21.15.148:0/1256892192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe000060280 0x7fe0001066e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.579+0000 7fe007844700 1 -- 172.21.15.148:0/1256892192 >> 172.21.15.148:0/1256892192 conn(0x7fe0000f0b70 msgr2=0x7fe0000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:31.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.579+0000 7fe007844700 1 -- 172.21.15.148:0/1256892192 shutdown_connections 2023-12-21T03:41:31.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:31.579+0000 7fe007844700 1 -- 172.21.15.148:0/1256892192 wait complete. 2023-12-21T03:41:31.581 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:30.541408+0000 mon.smithi078 (mon.0) 160 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:41:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:30.549537+0000 mon.smithi078 (mon.0) 161 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:41:32.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:31.578266+0000 mon.smithi078 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.148:0/1256892192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:32.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:32.419883+0000 mgr.smithi078.zidbgq (mgr.14162) 66 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:32.995 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:32.995 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:34.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.127+0000 7f546200d700 1 -- 172.21.15.148:0/157763707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f545c0f51a0 msgr2=0x7f545c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:34.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.127+0000 7f546200d700 1 --2- 172.21.15.148:0/157763707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f545c0f51a0 0x7f545c0f55c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f5458009fb0 tx=0x7f5458009b30).stop 2023-12-21T03:41:34.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.127+0000 7f546200d700 1 -- 172.21.15.148:0/157763707 shutdown_connections 2023-12-21T03:41:34.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.127+0000 7f546200d700 1 --2- 172.21.15.148:0/157763707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f545c0f51a0 0x7f545c0f55c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:34.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.127+0000 7f546200d700 1 -- 172.21.15.148:0/157763707 >> 172.21.15.148:0/157763707 conn(0x7f545c0f09a0 msgr2=0x7f545c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:34.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.127+0000 7f546200d700 1 -- 172.21.15.148:0/157763707 shutdown_connections 2023-12-21T03:41:34.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.127+0000 7f546200d700 1 -- 172.21.15.148:0/157763707 wait complete. 2023-12-21T03:41:34.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.127+0000 7f546200d700 1 Processor -- start 2023-12-21T03:41:34.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.128+0000 7f546200d700 1 -- start start 2023-12-21T03:41:34.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.128+0000 7f546200d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f545c0f51a0 0x7f545c109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:34.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.128+0000 7f546200d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545c060280 con 0x7f545c0f51a0 2023-12-21T03:41:34.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.128+0000 7f546100b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f545c0f51a0 0x7f545c109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:34.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.128+0000 7f546100b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f545c0f51a0 0x7f545c109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:47364/0 (socket says 172.21.15.148:47364) 2023-12-21T03:41:34.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.128+0000 7f546100b700 1 -- 172.21.15.148:0/4038109380 learned_addr learned my addr 172.21.15.148:0/4038109380 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:34.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.128+0000 7f546100b700 1 -- 172.21.15.148:0/4038109380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5458005040 con 0x7f545c0f51a0 2023-12-21T03:41:34.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.128+0000 7f546100b700 1 --2- 172.21.15.148:0/4038109380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f545c0f51a0 0x7f545c109170 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5458009830 tx=0x7f545800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:34.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.128+0000 7f5451ffb700 1 -- 172.21.15.148:0/4038109380 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5458009320 con 0x7f545c0f51a0 2023-12-21T03:41:34.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.128+0000 7f546200d700 1 -- 172.21.15.148:0/4038109380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545c060480 con 0x7f545c0f51a0 2023-12-21T03:41:34.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.129+0000 7f5451ffb700 1 -- 172.21.15.148:0/4038109380 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54580136c0 con 0x7f545c0f51a0 2023-12-21T03:41:34.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.129+0000 7f5451ffb700 1 -- 172.21.15.148:0/4038109380 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f545801ee90 con 0x7f545c0f51a0 2023-12-21T03:41:34.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.129+0000 7f546200d700 1 -- 172.21.15.148:0/4038109380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f545c1098a0 con 0x7f545c0f51a0 2023-12-21T03:41:34.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.130+0000 7f546200d700 1 -- 172.21.15.148:0/4038109380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545c045ad0 con 0x7f545c0f51a0 2023-12-21T03:41:34.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.130+0000 7f5451ffb700 1 -- 172.21.15.148:0/4038109380 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f545801e420 con 0x7f545c0f51a0 2023-12-21T03:41:34.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.130+0000 7f5451ffb700 1 --2- 172.21.15.148:0/4038109380 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f544802ff10 0x7f54480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:34.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.130+0000 7f5451ffb700 1 -- 172.21.15.148:0/4038109380 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5458040f40 con 0x7f545c0f51a0 2023-12-21T03:41:34.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.130+0000 7f546080a700 1 --2- 172.21.15.148:0/4038109380 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f544802ff10 0x7f54480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:34.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.130+0000 7f546080a700 1 --2- 172.21.15.148:0/4038109380 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f544802ff10 0x7f54480323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f544c0096a0 tx=0x7f544c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:34.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.135+0000 7f5451ffb700 1 -- 172.21.15.148:0/4038109380 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f545806d2b0 con 0x7f545c0f51a0 2023-12-21T03:41:34.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.341+0000 7f546200d700 1 -- 172.21.15.148:0/4038109380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f545c0fabe0 con 0x7f545c0f51a0 2023-12-21T03:41:34.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.341+0000 7f5451ffb700 1 -- 172.21.15.148:0/4038109380 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f54580095f0 con 0x7f545c0f51a0 2023-12-21T03:41:34.343 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:34.343 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:34.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.343+0000 7f546200d700 1 -- 172.21.15.148:0/4038109380 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f544802ff10 msgr2=0x7f54480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:34.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.343+0000 7f546200d700 1 --2- 172.21.15.148:0/4038109380 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f544802ff10 0x7f54480323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f544c0096a0 tx=0x7f544c006b40).stop 2023-12-21T03:41:34.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.343+0000 7f546200d700 1 -- 172.21.15.148:0/4038109380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f545c0f51a0 msgr2=0x7f545c109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:34.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.343+0000 7f546200d700 1 --2- 172.21.15.148:0/4038109380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f545c0f51a0 0x7f545c109170 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5458009830 tx=0x7f545800b040).stop 2023-12-21T03:41:34.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.343+0000 7f546200d700 1 -- 172.21.15.148:0/4038109380 shutdown_connections 2023-12-21T03:41:34.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.343+0000 7f546200d700 1 --2- 172.21.15.148:0/4038109380 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f544802ff10 0x7f54480323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:34.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.343+0000 7f546200d700 1 --2- 172.21.15.148:0/4038109380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f545c0f51a0 0x7f545c109170 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:34.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.343+0000 7f546200d700 1 -- 172.21.15.148:0/4038109380 >> 172.21.15.148:0/4038109380 conn(0x7f545c0f09a0 msgr2=0x7f545c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:34.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.343+0000 7f546200d700 1 -- 172.21.15.148:0/4038109380 shutdown_connections 2023-12-21T03:41:34.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:34.343+0000 7f546200d700 1 -- 172.21.15.148:0/4038109380 wait complete. 2023-12-21T03:41:34.345 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:35.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:34.342237+0000 mon.smithi078 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.148:0/4038109380' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:35.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:34.420268+0000 mgr.smithi078.zidbgq (mgr.14162) 67 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:35.717 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:35.717 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:36.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:36 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:36.420582+0000 mgr.smithi078.zidbgq (mgr.14162) 68 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:36.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.886+0000 7fbadb17f700 1 -- 172.21.15.148:0/2537699233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbad40f47e0 msgr2=0x7fbad40f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:36.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.886+0000 7fbadb17f700 1 --2- 172.21.15.148:0/2537699233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbad40f47e0 0x7fbad40f4c00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fbac8009fb0 tx=0x7fbac8009b30).stop 2023-12-21T03:41:36.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.886+0000 7fbadb17f700 1 -- 172.21.15.148:0/2537699233 shutdown_connections 2023-12-21T03:41:36.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.886+0000 7fbadb17f700 1 --2- 172.21.15.148:0/2537699233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbad40f47e0 0x7fbad40f4c00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:36.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.886+0000 7fbadb17f700 1 -- 172.21.15.148:0/2537699233 >> 172.21.15.148:0/2537699233 conn(0x7fbad40f0260 msgr2=0x7fbad40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:36.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.886+0000 7fbadb17f700 1 -- 172.21.15.148:0/2537699233 shutdown_connections 2023-12-21T03:41:36.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.886+0000 7fbadb17f700 1 -- 172.21.15.148:0/2537699233 wait complete. 2023-12-21T03:41:36.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.887+0000 7fbadb17f700 1 Processor -- start 2023-12-21T03:41:36.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.887+0000 7fbadb17f700 1 -- start start 2023-12-21T03:41:36.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.888+0000 7fbadb17f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbad40f47e0 0x7fbad41068d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:36.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.888+0000 7fbadb17f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad4106dc0 con 0x7fbad40f47e0 2023-12-21T03:41:36.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.888+0000 7fbad8f1b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbad40f47e0 0x7fbad41068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:36.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.888+0000 7fbad8f1b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbad40f47e0 0x7fbad41068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:47378/0 (socket says 172.21.15.148:47378) 2023-12-21T03:41:36.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.888+0000 7fbad8f1b700 1 -- 172.21.15.148:0/1466422665 learned_addr learned my addr 172.21.15.148:0/1466422665 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:36.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.888+0000 7fbad8f1b700 1 -- 172.21.15.148:0/1466422665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac8005040 con 0x7fbad40f47e0 2023-12-21T03:41:36.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.888+0000 7fbad8f1b700 1 --2- 172.21.15.148:0/1466422665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbad40f47e0 0x7fbad41068d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fbac8013510 tx=0x7fbac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:36.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.889+0000 7fbad17fa700 1 -- 172.21.15.148:0/1466422665 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbac8009320 con 0x7fbad40f47e0 2023-12-21T03:41:36.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.889+0000 7fbadb17f700 1 -- 172.21.15.148:0/1466422665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad4106fc0 con 0x7fbad40f47e0 2023-12-21T03:41:36.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.889+0000 7fbad17fa700 1 -- 172.21.15.148:0/1466422665 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbac8013f00 con 0x7fbad40f47e0 2023-12-21T03:41:36.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.889+0000 7fbad17fa700 1 -- 172.21.15.148:0/1466422665 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbac8014dc0 con 0x7fbad40f47e0 2023-12-21T03:41:36.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.889+0000 7fbadb17f700 1 -- 172.21.15.148:0/1466422665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad4107450 con 0x7fbad40f47e0 2023-12-21T03:41:36.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.889+0000 7fbad17fa700 1 -- 172.21.15.148:0/1466422665 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbac801d3d0 con 0x7fbad40f47e0 2023-12-21T03:41:36.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.889+0000 7fbadb17f700 1 -- 172.21.15.148:0/1466422665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad4045ad0 con 0x7fbad40f47e0 2023-12-21T03:41:36.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.890+0000 7fbad17fa700 1 --2- 172.21.15.148:0/1466422665 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbac402ff60 0x7fbac4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:36.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.890+0000 7fbad17fa700 1 -- 172.21.15.148:0/1466422665 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbac8041300 con 0x7fbad40f47e0 2023-12-21T03:41:36.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.890+0000 7fbad3fff700 1 --2- 172.21.15.148:0/1466422665 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbac402ff60 0x7fbac4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:36.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.890+0000 7fbad3fff700 1 --2- 172.21.15.148:0/1466422665 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbac402ff60 0x7fbac4032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fbac00096a0 tx=0x7fbac0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:36.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:36.894+0000 7fbad17fa700 1 -- 172.21.15.148:0/1466422665 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbac801cad0 con 0x7fbad40f47e0 2023-12-21T03:41:37.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.100+0000 7fbadb17f700 1 -- 172.21.15.148:0/1466422665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbad405eb50 con 0x7fbad40f47e0 2023-12-21T03:41:37.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.101+0000 7fbad17fa700 1 -- 172.21.15.148:0/1466422665 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbac801cad0 con 0x7fbad40f47e0 2023-12-21T03:41:37.103 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:37.103 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:37.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.103+0000 7fbadb17f700 1 -- 172.21.15.148:0/1466422665 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbac402ff60 msgr2=0x7fbac4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:37.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.103+0000 7fbadb17f700 1 --2- 172.21.15.148:0/1466422665 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbac402ff60 0x7fbac4032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fbac00096a0 tx=0x7fbac0006b40).stop 2023-12-21T03:41:37.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.103+0000 7fbadb17f700 1 -- 172.21.15.148:0/1466422665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbad40f47e0 msgr2=0x7fbad41068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:37.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.103+0000 7fbadb17f700 1 --2- 172.21.15.148:0/1466422665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbad40f47e0 0x7fbad41068d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fbac8013510 tx=0x7fbac800b040).stop 2023-12-21T03:41:37.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.103+0000 7fbadb17f700 1 -- 172.21.15.148:0/1466422665 shutdown_connections 2023-12-21T03:41:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.103+0000 7fbadb17f700 1 --2- 172.21.15.148:0/1466422665 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbac402ff60 0x7fbac4032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.103+0000 7fbadb17f700 1 --2- 172.21.15.148:0/1466422665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbad40f47e0 0x7fbad41068d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.103+0000 7fbadb17f700 1 -- 172.21.15.148:0/1466422665 >> 172.21.15.148:0/1466422665 conn(0x7fbad40f0260 msgr2=0x7fbad40f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.103+0000 7fbadb17f700 1 -- 172.21.15.148:0/1466422665 shutdown_connections 2023-12-21T03:41:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:37.103+0000 7fbadb17f700 1 -- 172.21.15.148:0/1466422665 wait complete. 2023-12-21T03:41:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:37.101913+0000 mon.smithi078 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.148:0/1466422665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:38.469 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:38.469 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:38.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:38.420979+0000 mgr.smithi078.zidbgq (mgr.14162) 69 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:39.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.599+0000 7f85fb9da700 1 -- 172.21.15.148:0/2831074084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f40f5410 msgr2=0x7f85f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:39.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.599+0000 7f85fb9da700 1 --2- 172.21.15.148:0/2831074084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f40f5410 0x7f85f40f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f85e4009fb0 tx=0x7f85e4009b30).stop 2023-12-21T03:41:39.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.599+0000 7f85fb9da700 1 -- 172.21.15.148:0/2831074084 shutdown_connections 2023-12-21T03:41:39.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.599+0000 7f85fb9da700 1 --2- 172.21.15.148:0/2831074084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f40f5410 0x7f85f40f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:39.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.599+0000 7f85fb9da700 1 -- 172.21.15.148:0/2831074084 >> 172.21.15.148:0/2831074084 conn(0x7f85f40f0b70 msgr2=0x7f85f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:39.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.599+0000 7f85fb9da700 1 -- 172.21.15.148:0/2831074084 shutdown_connections 2023-12-21T03:41:39.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.599+0000 7f85fb9da700 1 -- 172.21.15.148:0/2831074084 wait complete. 2023-12-21T03:41:39.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.599+0000 7f85fb9da700 1 Processor -- start 2023-12-21T03:41:39.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.599+0000 7f85fb9da700 1 -- start start 2023-12-21T03:41:39.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.600+0000 7f85fb9da700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f40f5410 0x7f85f4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:39.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.600+0000 7f85fb9da700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4105b20 con 0x7f85f40f5410 2023-12-21T03:41:39.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.600+0000 7f85f9776700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f40f5410 0x7f85f4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:39.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.600+0000 7f85f9776700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f40f5410 0x7f85f4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:54856/0 (socket says 172.21.15.148:54856) 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.600+0000 7f85f9776700 1 -- 172.21.15.148:0/808328468 learned_addr learned my addr 172.21.15.148:0/808328468 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.600+0000 7f85f9776700 1 -- 172.21.15.148:0/808328468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e4005040 con 0x7f85f40f5410 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.600+0000 7f85f9776700 1 --2- 172.21.15.148:0/808328468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f40f5410 0x7f85f4105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f85e4013510 tx=0x7f85e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.600+0000 7f85e9ffb700 1 -- 172.21.15.148:0/808328468 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85e4009320 con 0x7f85f40f5410 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.601+0000 7f85fb9da700 1 -- 172.21.15.148:0/808328468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f4105d20 con 0x7f85f40f5410 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.601+0000 7f85e9ffb700 1 -- 172.21.15.148:0/808328468 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85e4013f00 con 0x7f85f40f5410 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.601+0000 7f85e9ffb700 1 -- 172.21.15.148:0/808328468 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85e4014d90 con 0x7f85f40f5410 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.601+0000 7f85fb9da700 1 -- 172.21.15.148:0/808328468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f41061b0 con 0x7f85f40f5410 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.601+0000 7f85e9ffb700 1 -- 172.21.15.148:0/808328468 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85e40095f0 con 0x7f85f40f5410 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.602+0000 7f85fb9da700 1 -- 172.21.15.148:0/808328468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f4045ad0 con 0x7f85f40f5410 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.602+0000 7f85e9ffb700 1 --2- 172.21.15.148:0/808328468 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f85e002ff60 0x7f85e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.602+0000 7f85e9ffb700 1 -- 172.21.15.148:0/808328468 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85e40410c0 con 0x7f85f40f5410 2023-12-21T03:41:39.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.602+0000 7f85f8f75700 1 --2- 172.21.15.148:0/808328468 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f85e002ff60 0x7f85e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:39.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.602+0000 7f85f8f75700 1 --2- 172.21.15.148:0/808328468 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f85e002ff60 0x7f85e0032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f85f00096a0 tx=0x7f85f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:39.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.607+0000 7f85e9ffb700 1 -- 172.21.15.148:0/808328468 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85e401d7f0 con 0x7f85f40f5410 2023-12-21T03:41:39.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.810+0000 7f85fb9da700 1 -- 172.21.15.148:0/808328468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85f40fae60 con 0x7f85f40f5410 2023-12-21T03:41:39.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.811+0000 7f85e9ffb700 1 -- 172.21.15.148:0/808328468 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f85e4010230 con 0x7f85f40f5410 2023-12-21T03:41:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:39.813 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:39.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.812+0000 7f85fb9da700 1 -- 172.21.15.148:0/808328468 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f85e002ff60 msgr2=0x7f85e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:39.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.812+0000 7f85fb9da700 1 --2- 172.21.15.148:0/808328468 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f85e002ff60 0x7f85e0032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f85f00096a0 tx=0x7f85f0006b40).stop 2023-12-21T03:41:39.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.812+0000 7f85fb9da700 1 -- 172.21.15.148:0/808328468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f40f5410 msgr2=0x7f85f4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:39.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.812+0000 7f85fb9da700 1 --2- 172.21.15.148:0/808328468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f40f5410 0x7f85f4105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f85e4013510 tx=0x7f85e400b040).stop 2023-12-21T03:41:39.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.812+0000 7f85fb9da700 1 -- 172.21.15.148:0/808328468 shutdown_connections 2023-12-21T03:41:39.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.812+0000 7f85fb9da700 1 --2- 172.21.15.148:0/808328468 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f85e002ff60 0x7f85e0032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:39.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.812+0000 7f85fb9da700 1 --2- 172.21.15.148:0/808328468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f40f5410 0x7f85f4105630 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:39.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.812+0000 7f85fb9da700 1 -- 172.21.15.148:0/808328468 >> 172.21.15.148:0/808328468 conn(0x7f85f40f0b70 msgr2=0x7f85f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:39.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.812+0000 7f85fb9da700 1 -- 172.21.15.148:0/808328468 shutdown_connections 2023-12-21T03:41:39.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:39.812+0000 7f85fb9da700 1 -- 172.21.15.148:0/808328468 wait complete. 2023-12-21T03:41:39.816 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:40.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:39.812543+0000 mon.smithi078 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.148:0/808328468' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:41.201 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:41.201 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:41.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:40.421369+0000 mgr.smithi078.zidbgq (mgr.14162) 70 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:42.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.318+0000 7f0849728700 1 -- 172.21.15.148:0/2097266335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08440f5410 msgr2=0x7f08440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:42.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.318+0000 7f0849728700 1 --2- 172.21.15.148:0/2097266335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08440f5410 0x7f08440f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f0838009fb0 tx=0x7f0838009b30).stop 2023-12-21T03:41:42.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.318+0000 7f0849728700 1 -- 172.21.15.148:0/2097266335 shutdown_connections 2023-12-21T03:41:42.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.318+0000 7f0849728700 1 --2- 172.21.15.148:0/2097266335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08440f5410 0x7f08440f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:42.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.318+0000 7f0849728700 1 -- 172.21.15.148:0/2097266335 >> 172.21.15.148:0/2097266335 conn(0x7f08440f0b70 msgr2=0x7f08440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:42.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.318+0000 7f0849728700 1 -- 172.21.15.148:0/2097266335 shutdown_connections 2023-12-21T03:41:42.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.318+0000 7f0849728700 1 -- 172.21.15.148:0/2097266335 wait complete. 2023-12-21T03:41:42.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.319+0000 7f0849728700 1 Processor -- start 2023-12-21T03:41:42.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.319+0000 7f0849728700 1 -- start start 2023-12-21T03:41:42.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.319+0000 7f0849728700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08440f5410 0x7f0844105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:42.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.319+0000 7f0849728700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0844105af0 con 0x7f08440f5410 2023-12-21T03:41:42.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.319+0000 7f0842ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08440f5410 0x7f0844105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:42.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.319+0000 7f0842ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08440f5410 0x7f0844105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:54874/0 (socket says 172.21.15.148:54874) 2023-12-21T03:41:42.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.319+0000 7f0842ffd700 1 -- 172.21.15.148:0/1405872009 learned_addr learned my addr 172.21.15.148:0/1405872009 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:42.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.319+0000 7f0842ffd700 1 -- 172.21.15.148:0/1405872009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0838005040 con 0x7f08440f5410 2023-12-21T03:41:42.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.320+0000 7f0842ffd700 1 --2- 172.21.15.148:0/1405872009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08440f5410 0x7f0844105600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f0838013510 tx=0x7f083800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:42.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.320+0000 7f082b7fe700 1 -- 172.21.15.148:0/1405872009 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0838009320 con 0x7f08440f5410 2023-12-21T03:41:42.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.320+0000 7f082b7fe700 1 -- 172.21.15.148:0/1405872009 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0838013f00 con 0x7f08440f5410 2023-12-21T03:41:42.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.320+0000 7f0849728700 1 -- 172.21.15.148:0/1405872009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0844105cf0 con 0x7f08440f5410 2023-12-21T03:41:42.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.320+0000 7f082b7fe700 1 -- 172.21.15.148:0/1405872009 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0838014d90 con 0x7f08440f5410 2023-12-21T03:41:42.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.320+0000 7f0849728700 1 -- 172.21.15.148:0/1405872009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0844106180 con 0x7f08440f5410 2023-12-21T03:41:42.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.321+0000 7f082b7fe700 1 -- 172.21.15.148:0/1405872009 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0838014420 con 0x7f08440f5410 2023-12-21T03:41:42.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.321+0000 7f0849728700 1 -- 172.21.15.148:0/1405872009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0844045ad0 con 0x7f08440f5410 2023-12-21T03:41:42.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.321+0000 7f082b7fe700 1 --2- 172.21.15.148:0/1405872009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f082c02ffb0 0x7f082c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:42.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.321+0000 7f082b7fe700 1 -- 172.21.15.148:0/1405872009 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0838041090 con 0x7f08440f5410 2023-12-21T03:41:42.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.321+0000 7f08427fc700 1 --2- 172.21.15.148:0/1405872009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f082c02ffb0 0x7f082c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:42.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.322+0000 7f08427fc700 1 --2- 172.21.15.148:0/1405872009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f082c02ffb0 0x7f082c032470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f08340096a0 tx=0x7f0834006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:42.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.326+0000 7f082b7fe700 1 -- 172.21.15.148:0/1405872009 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f083806d2b0 con 0x7f08440f5410 2023-12-21T03:41:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.533+0000 7f0849728700 1 -- 172.21.15.148:0/1405872009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08440fae60 con 0x7f08440f5410 2023-12-21T03:41:42.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.534+0000 7f082b7fe700 1 -- 172.21.15.148:0/1405872009 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f08380095f0 con 0x7f08440f5410 2023-12-21T03:41:42.536 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:42.536 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.536+0000 7f0849728700 1 -- 172.21.15.148:0/1405872009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f082c02ffb0 msgr2=0x7f082c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.536+0000 7f0849728700 1 --2- 172.21.15.148:0/1405872009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f082c02ffb0 0x7f082c032470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f08340096a0 tx=0x7f0834006b40).stop 2023-12-21T03:41:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.536+0000 7f0849728700 1 -- 172.21.15.148:0/1405872009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08440f5410 msgr2=0x7f0844105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.536+0000 7f0849728700 1 --2- 172.21.15.148:0/1405872009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08440f5410 0x7f0844105600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f0838013510 tx=0x7f083800b040).stop 2023-12-21T03:41:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.536+0000 7f0849728700 1 -- 172.21.15.148:0/1405872009 shutdown_connections 2023-12-21T03:41:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.536+0000 7f0849728700 1 --2- 172.21.15.148:0/1405872009 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f082c02ffb0 0x7f082c032470 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.536+0000 7f0849728700 1 --2- 172.21.15.148:0/1405872009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08440f5410 0x7f0844105600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.536+0000 7f0849728700 1 -- 172.21.15.148:0/1405872009 >> 172.21.15.148:0/1405872009 conn(0x7f08440f0b70 msgr2=0x7f08440f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.536+0000 7f0849728700 1 -- 172.21.15.148:0/1405872009 shutdown_connections 2023-12-21T03:41:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:42.536+0000 7f0849728700 1 -- 172.21.15.148:0/1405872009 wait complete. 2023-12-21T03:41:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:42.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:42.421701+0000 mgr.smithi078.zidbgq (mgr.14162) 71 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:42.535905+0000 mon.smithi078 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.148:0/1405872009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:43.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:43.942 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:45.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.124+0000 7f9823537700 1 -- 172.21.15.148:0/2222290797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f981c0f5410 msgr2=0x7f981c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:45.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.124+0000 7f9823537700 1 --2- 172.21.15.148:0/2222290797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f981c0f5410 0x7f981c0f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f980c009fb0 tx=0x7f980c009b30).stop 2023-12-21T03:41:45.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.124+0000 7f9823537700 1 -- 172.21.15.148:0/2222290797 shutdown_connections 2023-12-21T03:41:45.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.124+0000 7f9823537700 1 --2- 172.21.15.148:0/2222290797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f981c0f5410 0x7f981c0f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:45.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.124+0000 7f9823537700 1 -- 172.21.15.148:0/2222290797 >> 172.21.15.148:0/2222290797 conn(0x7f981c0f0b70 msgr2=0x7f981c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:45.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.124+0000 7f9823537700 1 -- 172.21.15.148:0/2222290797 shutdown_connections 2023-12-21T03:41:45.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.124+0000 7f9823537700 1 -- 172.21.15.148:0/2222290797 wait complete. 2023-12-21T03:41:45.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.125+0000 7f9823537700 1 Processor -- start 2023-12-21T03:41:45.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.125+0000 7f9823537700 1 -- start start 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.125+0000 7f9823537700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f981c0f5410 0x7f981c10b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.125+0000 7f9823537700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c107210 con 0x7f981c0f5410 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.125+0000 7f98212d3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f981c0f5410 0x7f981c10b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.125+0000 7f98212d3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f981c0f5410 0x7f981c10b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:54894/0 (socket says 172.21.15.148:54894) 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.125+0000 7f98212d3700 1 -- 172.21.15.148:0/1683064848 learned_addr learned my addr 172.21.15.148:0/1683064848 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.125+0000 7f98212d3700 1 -- 172.21.15.148:0/1683064848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c005040 con 0x7f981c0f5410 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.126+0000 7f98212d3700 1 --2- 172.21.15.148:0/1683064848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f981c0f5410 0x7f981c10b820 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f980c013510 tx=0x7f980c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.126+0000 7f9811ffb700 1 -- 172.21.15.148:0/1683064848 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f980c009320 con 0x7f981c0f5410 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.126+0000 7f9823537700 1 -- 172.21.15.148:0/1683064848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c060220 con 0x7f981c0f5410 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.126+0000 7f9811ffb700 1 -- 172.21.15.148:0/1683064848 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f980c013f00 con 0x7f981c0f5410 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.126+0000 7f9811ffb700 1 -- 172.21.15.148:0/1683064848 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f980c014d90 con 0x7f981c0f5410 2023-12-21T03:41:45.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.126+0000 7f9823537700 1 -- 172.21.15.148:0/1683064848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c10bd10 con 0x7f981c0f5410 2023-12-21T03:41:45.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.127+0000 7f9823537700 1 -- 172.21.15.148:0/1683064848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c045ad0 con 0x7f981c0f5410 2023-12-21T03:41:45.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.127+0000 7f9811ffb700 1 -- 172.21.15.148:0/1683064848 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f980c0095f0 con 0x7f981c0f5410 2023-12-21T03:41:45.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.127+0000 7f9811ffb700 1 --2- 172.21.15.148:0/1683064848 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f980802ff60 0x7f9808032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:45.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.127+0000 7f9811ffb700 1 -- 172.21.15.148:0/1683064848 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f980c040fe0 con 0x7f981c0f5410 2023-12-21T03:41:45.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.127+0000 7f9820ad2700 1 --2- 172.21.15.148:0/1683064848 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f980802ff60 0x7f9808032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:45.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.128+0000 7f9820ad2700 1 --2- 172.21.15.148:0/1683064848 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f980802ff60 0x7f9808032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f98180096a0 tx=0x7f9818006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:45.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.132+0000 7f9811ffb700 1 -- 172.21.15.148:0/1683064848 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f980c01d7f0 con 0x7f981c0f5410 2023-12-21T03:41:45.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.338+0000 7f9823537700 1 -- 172.21.15.148:0/1683064848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f981c0fae60 con 0x7f981c0f5410 2023-12-21T03:41:45.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.338+0000 7f9811ffb700 1 -- 172.21.15.148:0/1683064848 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f980c010230 con 0x7f981c0f5410 2023-12-21T03:41:45.341 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:45.341 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:45.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.340+0000 7f9823537700 1 -- 172.21.15.148:0/1683064848 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f980802ff60 msgr2=0x7f9808032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:45.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.340+0000 7f9823537700 1 --2- 172.21.15.148:0/1683064848 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f980802ff60 0x7f9808032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f98180096a0 tx=0x7f9818006b40).stop 2023-12-21T03:41:45.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.340+0000 7f9823537700 1 -- 172.21.15.148:0/1683064848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f981c0f5410 msgr2=0x7f981c10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:45.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.340+0000 7f9823537700 1 --2- 172.21.15.148:0/1683064848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f981c0f5410 0x7f981c10b820 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f980c013510 tx=0x7f980c00b040).stop 2023-12-21T03:41:45.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.340+0000 7f9823537700 1 -- 172.21.15.148:0/1683064848 shutdown_connections 2023-12-21T03:41:45.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.340+0000 7f9823537700 1 --2- 172.21.15.148:0/1683064848 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f980802ff60 0x7f9808032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:45.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.340+0000 7f9823537700 1 --2- 172.21.15.148:0/1683064848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f981c0f5410 0x7f981c10b820 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:45.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.340+0000 7f9823537700 1 -- 172.21.15.148:0/1683064848 >> 172.21.15.148:0/1683064848 conn(0x7f981c0f0b70 msgr2=0x7f981c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:45.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.340+0000 7f9823537700 1 -- 172.21.15.148:0/1683064848 shutdown_connections 2023-12-21T03:41:45.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:45.340+0000 7f9823537700 1 -- 172.21.15.148:0/1683064848 wait complete. 2023-12-21T03:41:45.343 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:45.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:44.422074+0000 mgr.smithi078.zidbgq (mgr.14162) 72 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:46.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:46 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:45.340284+0000 mon.smithi078 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.148:0/1683064848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:46.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:46.670 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:47.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:46.422491+0000 mgr.smithi078.zidbgq (mgr.14162) 73 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.776+0000 7f7097fff700 1 -- 172.21.15.148:0/3409942295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70980f51a0 msgr2=0x7f70980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.776+0000 7f7097fff700 1 --2- 172.21.15.148:0/3409942295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70980f51a0 0x7f70980f55c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7088009fb0 tx=0x7f7088009b30).stop 2023-12-21T03:41:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.776+0000 7f7097fff700 1 -- 172.21.15.148:0/3409942295 shutdown_connections 2023-12-21T03:41:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.776+0000 7f7097fff700 1 --2- 172.21.15.148:0/3409942295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70980f51a0 0x7f70980f55c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.776+0000 7f7097fff700 1 -- 172.21.15.148:0/3409942295 >> 172.21.15.148:0/3409942295 conn(0x7f70980f09a0 msgr2=0x7f70980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.776+0000 7f7097fff700 1 -- 172.21.15.148:0/3409942295 shutdown_connections 2023-12-21T03:41:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.776+0000 7f7097fff700 1 -- 172.21.15.148:0/3409942295 wait complete. 2023-12-21T03:41:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.777+0000 7f7097fff700 1 Processor -- start 2023-12-21T03:41:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.777+0000 7f7097fff700 1 -- start start 2023-12-21T03:41:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.777+0000 7f7097fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70980f51a0 0x7f70981091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.777+0000 7f7097fff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7098060220 con 0x7f70980f51a0 2023-12-21T03:41:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.777+0000 7f7096ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70980f51a0 0x7f70981091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.777+0000 7f7096ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70980f51a0 0x7f70981091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:54920/0 (socket says 172.21.15.148:54920) 2023-12-21T03:41:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.777+0000 7f7096ffd700 1 -- 172.21.15.148:0/3248480298 learned_addr learned my addr 172.21.15.148:0/3248480298 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.778+0000 7f7096ffd700 1 -- 172.21.15.148:0/3248480298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7088005040 con 0x7f70980f51a0 2023-12-21T03:41:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.778+0000 7f7096ffd700 1 --2- 172.21.15.148:0/3248480298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70980f51a0 0x7f70981091d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7088009830 tx=0x7f708800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.778+0000 7f708f7fe700 1 -- 172.21.15.148:0/3248480298 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7088009320 con 0x7f70980f51a0 2023-12-21T03:41:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.778+0000 7f7097fff700 1 -- 172.21.15.148:0/3248480298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7098060420 con 0x7f70980f51a0 2023-12-21T03:41:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.778+0000 7f708f7fe700 1 -- 172.21.15.148:0/3248480298 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70880136c0 con 0x7f70980f51a0 2023-12-21T03:41:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.778+0000 7f708f7fe700 1 -- 172.21.15.148:0/3248480298 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f708801ee90 con 0x7f70980f51a0 2023-12-21T03:41:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.778+0000 7f7097fff700 1 -- 172.21.15.148:0/3248480298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7098109880 con 0x7f70980f51a0 2023-12-21T03:41:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.779+0000 7f708f7fe700 1 -- 172.21.15.148:0/3248480298 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f708801e420 con 0x7f70980f51a0 2023-12-21T03:41:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.779+0000 7f7097fff700 1 -- 172.21.15.148:0/3248480298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7098059100 con 0x7f70980f51a0 2023-12-21T03:41:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.779+0000 7f708f7fe700 1 --2- 172.21.15.148:0/3248480298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f708402ff10 0x7f70840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.779+0000 7f708f7fe700 1 -- 172.21.15.148:0/3248480298 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7088040f90 con 0x7f70980f51a0 2023-12-21T03:41:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.779+0000 7f70967fc700 1 --2- 172.21.15.148:0/3248480298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f708402ff10 0x7f70840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.779+0000 7f70967fc700 1 --2- 172.21.15.148:0/3248480298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f708402ff10 0x7f70840323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f70800096a0 tx=0x7f7080006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.784+0000 7f708f7fe700 1 -- 172.21.15.148:0/3248480298 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f708801c790 con 0x7f70980f51a0 2023-12-21T03:41:47.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.989+0000 7f7097fff700 1 -- 172.21.15.148:0/3248480298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7098045ad0 con 0x7f70980f51a0 2023-12-21T03:41:47.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.990+0000 7f708f7fe700 1 -- 172.21.15.148:0/3248480298 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f708802c2d0 con 0x7f70980f51a0 2023-12-21T03:41:47.992 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:47.993 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:47.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.992+0000 7f7097fff700 1 -- 172.21.15.148:0/3248480298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f708402ff10 msgr2=0x7f70840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:47.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.992+0000 7f7097fff700 1 --2- 172.21.15.148:0/3248480298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f708402ff10 0x7f70840323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f70800096a0 tx=0x7f7080006b40).stop 2023-12-21T03:41:47.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.992+0000 7f7097fff700 1 -- 172.21.15.148:0/3248480298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70980f51a0 msgr2=0x7f70981091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:47.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.992+0000 7f7097fff700 1 --2- 172.21.15.148:0/3248480298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70980f51a0 0x7f70981091d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7088009830 tx=0x7f708800b040).stop 2023-12-21T03:41:47.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.992+0000 7f7097fff700 1 -- 172.21.15.148:0/3248480298 shutdown_connections 2023-12-21T03:41:47.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.992+0000 7f7097fff700 1 --2- 172.21.15.148:0/3248480298 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f708402ff10 0x7f70840323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:47.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.992+0000 7f7097fff700 1 --2- 172.21.15.148:0/3248480298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70980f51a0 0x7f70981091d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:47.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.992+0000 7f7097fff700 1 -- 172.21.15.148:0/3248480298 >> 172.21.15.148:0/3248480298 conn(0x7f70980f09a0 msgr2=0x7f70980f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:47.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.992+0000 7f7097fff700 1 -- 172.21.15.148:0/3248480298 shutdown_connections 2023-12-21T03:41:47.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:47.992+0000 7f7097fff700 1 -- 172.21.15.148:0/3248480298 wait complete. 2023-12-21T03:41:47.995 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:48.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:47.991969+0000 mon.smithi078 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.148:0/3248480298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:49.371 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:49.371 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:49.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:48.422830+0000 mgr.smithi078.zidbgq (mgr.14162) 74 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:50.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.499+0000 7f8d604a3700 1 -- 172.21.15.148:0/2632958909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d580f6510 msgr2=0x7f8d580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:50.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.499+0000 7f8d604a3700 1 --2- 172.21.15.148:0/2632958909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d580f6510 0x7f8d580f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8d48009fb0 tx=0x7f8d48009b30).stop 2023-12-21T03:41:50.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.499+0000 7f8d604a3700 1 -- 172.21.15.148:0/2632958909 shutdown_connections 2023-12-21T03:41:50.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.499+0000 7f8d604a3700 1 --2- 172.21.15.148:0/2632958909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d580f6510 0x7f8d580f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:50.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.499+0000 7f8d604a3700 1 -- 172.21.15.148:0/2632958909 >> 172.21.15.148:0/2632958909 conn(0x7f8d580f1c70 msgr2=0x7f8d580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:50.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.499+0000 7f8d604a3700 1 -- 172.21.15.148:0/2632958909 shutdown_connections 2023-12-21T03:41:50.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.499+0000 7f8d604a3700 1 -- 172.21.15.148:0/2632958909 wait complete. 2023-12-21T03:41:50.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.499+0000 7f8d604a3700 1 Processor -- start 2023-12-21T03:41:50.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.499+0000 7f8d604a3700 1 -- start start 2023-12-21T03:41:50.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.500+0000 7f8d604a3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d580f6510 0x7f8d58106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:50.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.500+0000 7f8d604a3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d58106c20 con 0x7f8d580f6510 2023-12-21T03:41:50.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.500+0000 7f8d5e23f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d580f6510 0x7f8d58106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:50.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.500+0000 7f8d5e23f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d580f6510 0x7f8d58106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35244/0 (socket says 172.21.15.148:35244) 2023-12-21T03:41:50.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.500+0000 7f8d5e23f700 1 -- 172.21.15.148:0/3064777728 learned_addr learned my addr 172.21.15.148:0/3064777728 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:50.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.500+0000 7f8d5e23f700 1 -- 172.21.15.148:0/3064777728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d48005040 con 0x7f8d580f6510 2023-12-21T03:41:50.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.500+0000 7f8d5e23f700 1 --2- 172.21.15.148:0/3064777728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d580f6510 0x7f8d58106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8d48013510 tx=0x7f8d4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:50.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.500+0000 7f8d4effd700 1 -- 172.21.15.148:0/3064777728 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d48009320 con 0x7f8d580f6510 2023-12-21T03:41:50.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.500+0000 7f8d604a3700 1 -- 172.21.15.148:0/3064777728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d58106e20 con 0x7f8d580f6510 2023-12-21T03:41:50.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.500+0000 7f8d4effd700 1 -- 172.21.15.148:0/3064777728 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d48013f00 con 0x7f8d580f6510 2023-12-21T03:41:50.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.500+0000 7f8d4effd700 1 -- 172.21.15.148:0/3064777728 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d48014d90 con 0x7f8d580f6510 2023-12-21T03:41:50.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.501+0000 7f8d604a3700 1 -- 172.21.15.148:0/3064777728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d581072b0 con 0x7f8d580f6510 2023-12-21T03:41:50.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.501+0000 7f8d604a3700 1 -- 172.21.15.148:0/3064777728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d58045ad0 con 0x7f8d580f6510 2023-12-21T03:41:50.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.501+0000 7f8d4effd700 1 -- 172.21.15.148:0/3064777728 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d480095f0 con 0x7f8d580f6510 2023-12-21T03:41:50.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.501+0000 7f8d4effd700 1 --2- 172.21.15.148:0/3064777728 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4402ff60 0x7f8d44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:50.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.501+0000 7f8d4effd700 1 -- 172.21.15.148:0/3064777728 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d480410c0 con 0x7f8d580f6510 2023-12-21T03:41:50.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.502+0000 7f8d5da3e700 1 --2- 172.21.15.148:0/3064777728 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4402ff60 0x7f8d44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:50.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.502+0000 7f8d5da3e700 1 --2- 172.21.15.148:0/3064777728 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4402ff60 0x7f8d44032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8d540096a0 tx=0x7f8d54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:50.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.507+0000 7f8d4effd700 1 -- 172.21.15.148:0/3064777728 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d48014420 con 0x7f8d580f6510 2023-12-21T03:41:50.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.713+0000 7f8d604a3700 1 -- 172.21.15.148:0/3064777728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d580fbf60 con 0x7f8d580f6510 2023-12-21T03:41:50.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.715+0000 7f8d4effd700 1 -- 172.21.15.148:0/3064777728 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d48010230 con 0x7f8d580f6510 2023-12-21T03:41:50.716 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:50.716 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:50.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.716+0000 7f8d604a3700 1 -- 172.21.15.148:0/3064777728 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4402ff60 msgr2=0x7f8d44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:50.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.716+0000 7f8d604a3700 1 --2- 172.21.15.148:0/3064777728 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4402ff60 0x7f8d44032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8d540096a0 tx=0x7f8d54006b40).stop 2023-12-21T03:41:50.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.716+0000 7f8d604a3700 1 -- 172.21.15.148:0/3064777728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d580f6510 msgr2=0x7f8d58106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:50.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.716+0000 7f8d604a3700 1 --2- 172.21.15.148:0/3064777728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d580f6510 0x7f8d58106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8d48013510 tx=0x7f8d4800b040).stop 2023-12-21T03:41:50.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.716+0000 7f8d604a3700 1 -- 172.21.15.148:0/3064777728 shutdown_connections 2023-12-21T03:41:50.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.716+0000 7f8d604a3700 1 --2- 172.21.15.148:0/3064777728 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4402ff60 0x7f8d44032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:50.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.716+0000 7f8d604a3700 1 --2- 172.21.15.148:0/3064777728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d580f6510 0x7f8d58106730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:50.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.716+0000 7f8d604a3700 1 -- 172.21.15.148:0/3064777728 >> 172.21.15.148:0/3064777728 conn(0x7f8d580f1c70 msgr2=0x7f8d580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:50.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.716+0000 7f8d604a3700 1 -- 172.21.15.148:0/3064777728 shutdown_connections 2023-12-21T03:41:50.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:50.716+0000 7f8d604a3700 1 -- 172.21.15.148:0/3064777728 wait complete. 2023-12-21T03:41:50.719 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:50.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:50.423210+0000 mgr.smithi078.zidbgq (mgr.14162) 75 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:50.715962+0000 mon.smithi078 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.148:0/3064777728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:52.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:52.121 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:52.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:52.423600+0000 mgr.smithi078.zidbgq (mgr.14162) 76 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:53.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.270+0000 7f326df03700 1 -- 172.21.15.148:0/508029045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3268060280 msgr2=0x7f32680f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:53.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.270+0000 7f326df03700 1 --2- 172.21.15.148:0/508029045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3268060280 0x7f32680f90f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f3258009960 tx=0x7f3258004d10).stop 2023-12-21T03:41:53.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.270+0000 7f326df03700 1 -- 172.21.15.148:0/508029045 shutdown_connections 2023-12-21T03:41:53.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.270+0000 7f326df03700 1 --2- 172.21.15.148:0/508029045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3268060280 0x7f32680f90f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:53.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.270+0000 7f326df03700 1 -- 172.21.15.148:0/508029045 >> 172.21.15.148:0/508029045 conn(0x7f32680f0b70 msgr2=0x7f32680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:53.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.270+0000 7f326df03700 1 -- 172.21.15.148:0/508029045 shutdown_connections 2023-12-21T03:41:53.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.270+0000 7f326df03700 1 -- 172.21.15.148:0/508029045 wait complete. 2023-12-21T03:41:53.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.271+0000 7f326df03700 1 Processor -- start 2023-12-21T03:41:53.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.271+0000 7f326df03700 1 -- start start 2023-12-21T03:41:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.271+0000 7f326df03700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3268060280 0x7f32681066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.271+0000 7f326df03700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3268106bd0 con 0x7f3268060280 2023-12-21T03:41:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.272+0000 7f32677fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3268060280 0x7f32681066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.272+0000 7f32677fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3268060280 0x7f32681066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35258/0 (socket says 172.21.15.148:35258) 2023-12-21T03:41:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.272+0000 7f32677fe700 1 -- 172.21.15.148:0/3550188889 learned_addr learned my addr 172.21.15.148:0/3550188889 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.272+0000 7f32677fe700 1 -- 172.21.15.148:0/3550188889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3258009640 con 0x7f3268060280 2023-12-21T03:41:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.272+0000 7f32677fe700 1 --2- 172.21.15.148:0/3550188889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3268060280 0x7f32681066e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f3258007bd0 tx=0x7f3258004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:53.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.272+0000 7f325ffff700 1 -- 172.21.15.148:0/3550188889 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3258018070 con 0x7f3268060280 2023-12-21T03:41:53.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.272+0000 7f326df03700 1 -- 172.21.15.148:0/3550188889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3268106dd0 con 0x7f3268060280 2023-12-21T03:41:53.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.272+0000 7f325ffff700 1 -- 172.21.15.148:0/3550188889 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f325800b870 con 0x7f3268060280 2023-12-21T03:41:53.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.272+0000 7f325ffff700 1 -- 172.21.15.148:0/3550188889 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f325801d420 con 0x7f3268060280 2023-12-21T03:41:53.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.272+0000 7f326df03700 1 -- 172.21.15.148:0/3550188889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3268107260 con 0x7f3268060280 2023-12-21T03:41:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.273+0000 7f325ffff700 1 -- 172.21.15.148:0/3550188889 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3258024060 con 0x7f3268060280 2023-12-21T03:41:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.273+0000 7f325ffff700 1 --2- 172.21.15.148:0/3550188889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f325402ff60 0x7f3254032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.273+0000 7f325ffff700 1 -- 172.21.15.148:0/3550188889 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32580411b0 con 0x7f3268060280 2023-12-21T03:41:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.273+0000 7f326df03700 1 -- 172.21.15.148:0/3550188889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3268045ad0 con 0x7f3268060280 2023-12-21T03:41:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.273+0000 7f3266ffd700 1 --2- 172.21.15.148:0/3550188889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f325402ff60 0x7f3254032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.274+0000 7f3266ffd700 1 --2- 172.21.15.148:0/3550188889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f325402ff60 0x7f3254032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f32500096a0 tx=0x7f3250006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:53.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.278+0000 7f325ffff700 1 -- 172.21.15.148:0/3550188889 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3258011910 con 0x7f3268060280 2023-12-21T03:41:53.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.486+0000 7f326df03700 1 -- 172.21.15.148:0/3550188889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32680fc0a0 con 0x7f3268060280 2023-12-21T03:41:53.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.487+0000 7f325ffff700 1 -- 172.21.15.148:0/3550188889 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f32580048f0 con 0x7f3268060280 2023-12-21T03:41:53.488 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:53.489 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:53.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.489+0000 7f326df03700 1 -- 172.21.15.148:0/3550188889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f325402ff60 msgr2=0x7f3254032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:53.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.489+0000 7f326df03700 1 --2- 172.21.15.148:0/3550188889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f325402ff60 0x7f3254032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f32500096a0 tx=0x7f3250006b40).stop 2023-12-21T03:41:53.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.489+0000 7f326df03700 1 -- 172.21.15.148:0/3550188889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3268060280 msgr2=0x7f32681066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:53.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.489+0000 7f326df03700 1 --2- 172.21.15.148:0/3550188889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3268060280 0x7f32681066e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f3258007bd0 tx=0x7f3258004d10).stop 2023-12-21T03:41:53.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.489+0000 7f326df03700 1 -- 172.21.15.148:0/3550188889 shutdown_connections 2023-12-21T03:41:53.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.489+0000 7f326df03700 1 --2- 172.21.15.148:0/3550188889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f325402ff60 0x7f3254032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:53.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.489+0000 7f326df03700 1 --2- 172.21.15.148:0/3550188889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3268060280 0x7f32681066e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:53.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.489+0000 7f326df03700 1 -- 172.21.15.148:0/3550188889 >> 172.21.15.148:0/3550188889 conn(0x7f32680f0b70 msgr2=0x7f32680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:53.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.489+0000 7f326df03700 1 -- 172.21.15.148:0/3550188889 shutdown_connections 2023-12-21T03:41:53.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:53.489+0000 7f326df03700 1 -- 172.21.15.148:0/3550188889 wait complete. 2023-12-21T03:41:53.491 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:53.487780+0000 mon.smithi078 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.148:0/3550188889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:54.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:54.868 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:55.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:54.423952+0000 mgr.smithi078.zidbgq (mgr.14162) 77 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:56.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.041+0000 7fc495206700 1 -- 172.21.15.148:0/947706507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc490060280 msgr2=0x7fc4900f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:56.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.041+0000 7fc495206700 1 --2- 172.21.15.148:0/947706507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc490060280 0x7fc4900f90f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fc480009fb0 tx=0x7fc480009b30).stop 2023-12-21T03:41:56.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.041+0000 7fc495206700 1 -- 172.21.15.148:0/947706507 shutdown_connections 2023-12-21T03:41:56.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.041+0000 7fc495206700 1 --2- 172.21.15.148:0/947706507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc490060280 0x7fc4900f90f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:56.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.041+0000 7fc495206700 1 -- 172.21.15.148:0/947706507 >> 172.21.15.148:0/947706507 conn(0x7fc4900f0b70 msgr2=0x7fc4900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:56.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.041+0000 7fc495206700 1 -- 172.21.15.148:0/947706507 shutdown_connections 2023-12-21T03:41:56.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.041+0000 7fc495206700 1 -- 172.21.15.148:0/947706507 wait complete. 2023-12-21T03:41:56.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.041+0000 7fc495206700 1 Processor -- start 2023-12-21T03:41:56.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.041+0000 7fc495206700 1 -- start start 2023-12-21T03:41:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.042+0000 7fc495206700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc490060280 0x7fc4901066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.042+0000 7fc495206700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc490106bd0 con 0x7fc490060280 2023-12-21T03:41:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.042+0000 7fc48ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc490060280 0x7fc4901066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.042+0000 7fc48ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc490060280 0x7fc4901066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35274/0 (socket says 172.21.15.148:35274) 2023-12-21T03:41:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.042+0000 7fc48ed9d700 1 -- 172.21.15.148:0/3844077128 learned_addr learned my addr 172.21.15.148:0/3844077128 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.042+0000 7fc48ed9d700 1 -- 172.21.15.148:0/3844077128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc480005040 con 0x7fc490060280 2023-12-21T03:41:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.042+0000 7fc48ed9d700 1 --2- 172.21.15.148:0/3844077128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc490060280 0x7fc4901066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fc480013510 tx=0x7fc48000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.042+0000 7fc4777fe700 1 -- 172.21.15.148:0/3844077128 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc480009320 con 0x7fc490060280 2023-12-21T03:41:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.042+0000 7fc495206700 1 -- 172.21.15.148:0/3844077128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc490106dd0 con 0x7fc490060280 2023-12-21T03:41:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.042+0000 7fc4777fe700 1 -- 172.21.15.148:0/3844077128 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc480013f00 con 0x7fc490060280 2023-12-21T03:41:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.042+0000 7fc4777fe700 1 -- 172.21.15.148:0/3844077128 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc480014dc0 con 0x7fc490060280 2023-12-21T03:41:56.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.043+0000 7fc495206700 1 -- 172.21.15.148:0/3844077128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc490107260 con 0x7fc490060280 2023-12-21T03:41:56.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.043+0000 7fc495206700 1 -- 172.21.15.148:0/3844077128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc490045ad0 con 0x7fc490060280 2023-12-21T03:41:56.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.043+0000 7fc4777fe700 1 -- 172.21.15.148:0/3844077128 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4800095f0 con 0x7fc490060280 2023-12-21T03:41:56.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.044+0000 7fc4777fe700 1 --2- 172.21.15.148:0/3844077128 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc47802ffb0 0x7fc478032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:56.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.044+0000 7fc4777fe700 1 -- 172.21.15.148:0/3844077128 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc480040ff0 con 0x7fc490060280 2023-12-21T03:41:56.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.044+0000 7fc48e59c700 1 --2- 172.21.15.148:0/3844077128 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc47802ffb0 0x7fc478032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:56.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.044+0000 7fc48e59c700 1 --2- 172.21.15.148:0/3844077128 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc47802ffb0 0x7fc478032470 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fc4840096a0 tx=0x7fc484006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:56.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.048+0000 7fc4777fe700 1 -- 172.21.15.148:0/3844077128 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc48006d2b0 con 0x7fc490060280 2023-12-21T03:41:56.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.251+0000 7fc495206700 1 -- 172.21.15.148:0/3844077128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc490108110 con 0x7fc490060280 2023-12-21T03:41:56.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.252+0000 7fc4777fe700 1 -- 172.21.15.148:0/3844077128 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc480014420 con 0x7fc490060280 2023-12-21T03:41:56.253 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:56.253 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:56.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.253+0000 7fc495206700 1 -- 172.21.15.148:0/3844077128 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc47802ffb0 msgr2=0x7fc478032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:56.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.253+0000 7fc495206700 1 --2- 172.21.15.148:0/3844077128 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc47802ffb0 0x7fc478032470 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fc4840096a0 tx=0x7fc484006b40).stop 2023-12-21T03:41:56.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.253+0000 7fc495206700 1 -- 172.21.15.148:0/3844077128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc490060280 msgr2=0x7fc4901066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:56.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.253+0000 7fc495206700 1 --2- 172.21.15.148:0/3844077128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc490060280 0x7fc4901066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fc480013510 tx=0x7fc48000b040).stop 2023-12-21T03:41:56.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.253+0000 7fc495206700 1 -- 172.21.15.148:0/3844077128 shutdown_connections 2023-12-21T03:41:56.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.253+0000 7fc495206700 1 --2- 172.21.15.148:0/3844077128 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc47802ffb0 0x7fc478032470 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:56.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.253+0000 7fc495206700 1 --2- 172.21.15.148:0/3844077128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc490060280 0x7fc4901066e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:56.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.253+0000 7fc495206700 1 -- 172.21.15.148:0/3844077128 >> 172.21.15.148:0/3844077128 conn(0x7fc4900f0b70 msgr2=0x7fc4900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:56.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.253+0000 7fc495206700 1 -- 172.21.15.148:0/3844077128 shutdown_connections 2023-12-21T03:41:56.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:56.253+0000 7fc495206700 1 -- 172.21.15.148:0/3844077128 wait complete. 2023-12-21T03:41:56.255 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:56.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:56.252443+0000 mon.smithi078 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.148:0/3844077128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:41:57.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:56.424252+0000 mgr.smithi078.zidbgq (mgr.14162) 78 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:57.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:41:57.635 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:41:58.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.739+0000 7f91b2d6f700 1 -- 172.21.15.148:0/5113313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91ac0f5410 msgr2=0x7f91ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:58.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.739+0000 7f91b2d6f700 1 --2- 172.21.15.148:0/5113313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91ac0f5410 0x7f91ac0f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f91a0009fb0 tx=0x7f91a0009b30).stop 2023-12-21T03:41:58.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.740+0000 7f91b2d6f700 1 -- 172.21.15.148:0/5113313 shutdown_connections 2023-12-21T03:41:58.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.740+0000 7f91b2d6f700 1 --2- 172.21.15.148:0/5113313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91ac0f5410 0x7f91ac0f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:58.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.740+0000 7f91b2d6f700 1 -- 172.21.15.148:0/5113313 >> 172.21.15.148:0/5113313 conn(0x7f91ac0f0b70 msgr2=0x7f91ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:58.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.740+0000 7f91b2d6f700 1 -- 172.21.15.148:0/5113313 shutdown_connections 2023-12-21T03:41:58.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.740+0000 7f91b2d6f700 1 -- 172.21.15.148:0/5113313 wait complete. 2023-12-21T03:41:58.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.740+0000 7f91b2d6f700 1 Processor -- start 2023-12-21T03:41:58.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.740+0000 7f91b2d6f700 1 -- start start 2023-12-21T03:41:58.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.740+0000 7f91b2d6f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91ac0f5410 0x7f91ac106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:58.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.740+0000 7f91b2d6f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac106c10 con 0x7f91ac0f5410 2023-12-21T03:41:58.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.741+0000 7f91b0b0b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91ac0f5410 0x7f91ac106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:58.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.741+0000 7f91b0b0b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91ac0f5410 0x7f91ac106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35282/0 (socket says 172.21.15.148:35282) 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.741+0000 7f91b0b0b700 1 -- 172.21.15.148:0/3964530979 learned_addr learned my addr 172.21.15.148:0/3964530979 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.741+0000 7f91b0b0b700 1 -- 172.21.15.148:0/3964530979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a0005040 con 0x7f91ac0f5410 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.741+0000 7f91b0b0b700 1 --2- 172.21.15.148:0/3964530979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91ac0f5410 0x7f91ac106720 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f91a0013510 tx=0x7f91a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.741+0000 7f91a97fa700 1 -- 172.21.15.148:0/3964530979 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91a0009320 con 0x7f91ac0f5410 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.741+0000 7f91b2d6f700 1 -- 172.21.15.148:0/3964530979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ac106e10 con 0x7f91ac0f5410 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.741+0000 7f91a97fa700 1 -- 172.21.15.148:0/3964530979 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91a0013f00 con 0x7f91ac0f5410 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.741+0000 7f91a97fa700 1 -- 172.21.15.148:0/3964530979 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91a0014d90 con 0x7f91ac0f5410 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.741+0000 7f91b2d6f700 1 -- 172.21.15.148:0/3964530979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ac1072a0 con 0x7f91ac0f5410 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.742+0000 7f91a97fa700 1 -- 172.21.15.148:0/3964530979 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91a00263a0 con 0x7f91ac0f5410 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.742+0000 7f91a97fa700 1 --2- 172.21.15.148:0/3964530979 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f919c02ff60 0x7f919c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.742+0000 7f91a97fa700 1 -- 172.21.15.148:0/3964530979 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f91a0041020 con 0x7f91ac0f5410 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.742+0000 7f91b2d6f700 1 -- 172.21.15.148:0/3964530979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ac045ad0 con 0x7f91ac0f5410 2023-12-21T03:41:58.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.743+0000 7f91abfff700 1 --2- 172.21.15.148:0/3964530979 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f919c02ff60 0x7f919c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:41:58.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.743+0000 7f91abfff700 1 --2- 172.21.15.148:0/3964530979 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f919c02ff60 0x7f919c032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f91980096a0 tx=0x7f9198006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:41:58.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.747+0000 7f91a97fa700 1 -- 172.21.15.148:0/3964530979 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91a0012920 con 0x7f91ac0f5410 2023-12-21T03:41:58.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:58 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:41:58.424457+0000 mgr.smithi078.zidbgq (mgr.14162) 79 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:41:58.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.951+0000 7f91b2d6f700 1 -- 172.21.15.148:0/3964530979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91ac0faf50 con 0x7f91ac0f5410 2023-12-21T03:41:58.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.951+0000 7f91a97fa700 1 -- 172.21.15.148:0/3964530979 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f91a002b020 con 0x7f91ac0f5410 2023-12-21T03:41:58.953 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:41:58.953 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:41:58.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.953+0000 7f91b2d6f700 1 -- 172.21.15.148:0/3964530979 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f919c02ff60 msgr2=0x7f919c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:58.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.953+0000 7f91b2d6f700 1 --2- 172.21.15.148:0/3964530979 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f919c02ff60 0x7f919c032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f91980096a0 tx=0x7f9198006b40).stop 2023-12-21T03:41:58.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.953+0000 7f91b2d6f700 1 -- 172.21.15.148:0/3964530979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91ac0f5410 msgr2=0x7f91ac106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:41:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.953+0000 7f91b2d6f700 1 --2- 172.21.15.148:0/3964530979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91ac0f5410 0x7f91ac106720 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f91a0013510 tx=0x7f91a000b040).stop 2023-12-21T03:41:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.953+0000 7f91b2d6f700 1 -- 172.21.15.148:0/3964530979 shutdown_connections 2023-12-21T03:41:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.953+0000 7f91b2d6f700 1 --2- 172.21.15.148:0/3964530979 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f919c02ff60 0x7f919c032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.953+0000 7f91b2d6f700 1 --2- 172.21.15.148:0/3964530979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91ac0f5410 0x7f91ac106720 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:41:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.953+0000 7f91b2d6f700 1 -- 172.21.15.148:0/3964530979 >> 172.21.15.148:0/3964530979 conn(0x7f91ac0f0b70 msgr2=0x7f91ac05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:41:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.953+0000 7f91b2d6f700 1 -- 172.21.15.148:0/3964530979 shutdown_connections 2023-12-21T03:41:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:41:58.953+0000 7f91b2d6f700 1 -- 172.21.15.148:0/3964530979 wait complete. 2023-12-21T03:41:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:41:59.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:41:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:41:58.952234+0000 mon.smithi078 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.148:0/3964530979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:00.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:00.330 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:00.424724+0000 mgr.smithi078.zidbgq (mgr.14162) 80 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:01.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.474+0000 7f3089908700 1 -- 172.21.15.148:0/1662955509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30840f6510 msgr2=0x7f30840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:01.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.474+0000 7f3089908700 1 --2- 172.21.15.148:0/1662955509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30840f6510 0x7f30840f6930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3078009fb0 tx=0x7f3078009b30).stop 2023-12-21T03:42:01.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.474+0000 7f3089908700 1 -- 172.21.15.148:0/1662955509 shutdown_connections 2023-12-21T03:42:01.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.474+0000 7f3089908700 1 --2- 172.21.15.148:0/1662955509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30840f6510 0x7f30840f6930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:01.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.474+0000 7f3089908700 1 -- 172.21.15.148:0/1662955509 >> 172.21.15.148:0/1662955509 conn(0x7f30840f1c70 msgr2=0x7f30840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:01.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.474+0000 7f3089908700 1 -- 172.21.15.148:0/1662955509 shutdown_connections 2023-12-21T03:42:01.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.474+0000 7f3089908700 1 -- 172.21.15.148:0/1662955509 wait complete. 2023-12-21T03:42:01.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.475+0000 7f3089908700 1 Processor -- start 2023-12-21T03:42:01.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.475+0000 7f3089908700 1 -- start start 2023-12-21T03:42:01.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.475+0000 7f3089908700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30840f6510 0x7f3084106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:01.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.475+0000 7f3089908700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3084106c20 con 0x7f30840f6510 2023-12-21T03:42:01.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.476+0000 7f3082ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30840f6510 0x7f3084106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:01.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.476+0000 7f3082ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30840f6510 0x7f3084106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:34526/0 (socket says 172.21.15.148:34526) 2023-12-21T03:42:01.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.476+0000 7f3082ffd700 1 -- 172.21.15.148:0/1803608485 learned_addr learned my addr 172.21.15.148:0/1803608485 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:01.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.476+0000 7f3082ffd700 1 -- 172.21.15.148:0/1803608485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3078005040 con 0x7f30840f6510 2023-12-21T03:42:01.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.476+0000 7f3082ffd700 1 --2- 172.21.15.148:0/1803608485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30840f6510 0x7f3084106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3078014000 tx=0x7f307800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:01.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.476+0000 7f306b7fe700 1 -- 172.21.15.148:0/1803608485 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30780095f0 con 0x7f30840f6510 2023-12-21T03:42:01.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.476+0000 7f3089908700 1 -- 172.21.15.148:0/1803608485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3084106e20 con 0x7f30840f6510 2023-12-21T03:42:01.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.476+0000 7f306b7fe700 1 -- 172.21.15.148:0/1803608485 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3078007e70 con 0x7f30840f6510 2023-12-21T03:42:01.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.476+0000 7f306b7fe700 1 -- 172.21.15.148:0/1803608485 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3078014ea0 con 0x7f30840f6510 2023-12-21T03:42:01.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.476+0000 7f3089908700 1 -- 172.21.15.148:0/1803608485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30841072b0 con 0x7f30840f6510 2023-12-21T03:42:01.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.477+0000 7f306b7fe700 1 -- 172.21.15.148:0/1803608485 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3078025070 con 0x7f30840f6510 2023-12-21T03:42:01.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.477+0000 7f3089908700 1 -- 172.21.15.148:0/1803608485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3084045ad0 con 0x7f30840f6510 2023-12-21T03:42:01.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.477+0000 7f306b7fe700 1 --2- 172.21.15.148:0/1803608485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f306c02ff60 0x7f306c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:01.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.477+0000 7f306b7fe700 1 -- 172.21.15.148:0/1803608485 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3078040a40 con 0x7f30840f6510 2023-12-21T03:42:01.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.478+0000 7f30827fc700 1 --2- 172.21.15.148:0/1803608485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f306c02ff60 0x7f306c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:01.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.478+0000 7f30827fc700 1 --2- 172.21.15.148:0/1803608485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f306c02ff60 0x7f306c032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f30740096a0 tx=0x7f3074006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:01.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.482+0000 7f306b7fe700 1 -- 172.21.15.148:0/1803608485 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30780079c0 con 0x7f30840f6510 2023-12-21T03:42:01.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.689+0000 7f3089908700 1 -- 172.21.15.148:0/1803608485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30840fbf60 con 0x7f30840f6510 2023-12-21T03:42:01.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.689+0000 7f306b7fe700 1 -- 172.21.15.148:0/1803608485 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f307800f300 con 0x7f30840f6510 2023-12-21T03:42:01.691 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:01.691 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:01.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.691+0000 7f3089908700 1 -- 172.21.15.148:0/1803608485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f306c02ff60 msgr2=0x7f306c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:01.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.691+0000 7f3089908700 1 --2- 172.21.15.148:0/1803608485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f306c02ff60 0x7f306c032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f30740096a0 tx=0x7f3074006b40).stop 2023-12-21T03:42:01.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.691+0000 7f3089908700 1 -- 172.21.15.148:0/1803608485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30840f6510 msgr2=0x7f3084106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:01.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.691+0000 7f3089908700 1 --2- 172.21.15.148:0/1803608485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30840f6510 0x7f3084106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3078014000 tx=0x7f307800b040).stop 2023-12-21T03:42:01.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.691+0000 7f3089908700 1 -- 172.21.15.148:0/1803608485 shutdown_connections 2023-12-21T03:42:01.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.691+0000 7f3089908700 1 --2- 172.21.15.148:0/1803608485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f306c02ff60 0x7f306c032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:01.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.691+0000 7f3089908700 1 --2- 172.21.15.148:0/1803608485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30840f6510 0x7f3084106730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:01.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.691+0000 7f3089908700 1 -- 172.21.15.148:0/1803608485 >> 172.21.15.148:0/1803608485 conn(0x7f30840f1c70 msgr2=0x7f30840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:01.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.691+0000 7f3089908700 1 -- 172.21.15.148:0/1803608485 shutdown_connections 2023-12-21T03:42:01.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:01.691+0000 7f3089908700 1 -- 172.21.15.148:0/1803608485 wait complete. 2023-12-21T03:42:01.694 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:01.690357+0000 mon.smithi078 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.148:0/1803608485' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:03.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:02.425058+0000 mgr.smithi078.zidbgq (mgr.14162) 81 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:03.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:03.108 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:04.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.247+0000 7fea12faa700 1 -- 172.21.15.148:0/168333869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea0c0f5410 msgr2=0x7fea0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:04.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.247+0000 7fea12faa700 1 --2- 172.21.15.148:0/168333869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea0c0f5410 0x7fea0c0f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fea00009fb0 tx=0x7fea00009b30).stop 2023-12-21T03:42:04.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.247+0000 7fea12faa700 1 -- 172.21.15.148:0/168333869 shutdown_connections 2023-12-21T03:42:04.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.247+0000 7fea12faa700 1 --2- 172.21.15.148:0/168333869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea0c0f5410 0x7fea0c0f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:04.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.247+0000 7fea12faa700 1 -- 172.21.15.148:0/168333869 >> 172.21.15.148:0/168333869 conn(0x7fea0c0f0b70 msgr2=0x7fea0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:04.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.247+0000 7fea12faa700 1 -- 172.21.15.148:0/168333869 shutdown_connections 2023-12-21T03:42:04.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.247+0000 7fea12faa700 1 -- 172.21.15.148:0/168333869 wait complete. 2023-12-21T03:42:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.247+0000 7fea12faa700 1 Processor -- start 2023-12-21T03:42:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.248+0000 7fea12faa700 1 -- start start 2023-12-21T03:42:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.248+0000 7fea12faa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea0c0f5410 0x7fea0c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.248+0000 7fea12faa700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c105b20 con 0x7fea0c0f5410 2023-12-21T03:42:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.248+0000 7fea10d46700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea0c0f5410 0x7fea0c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.248+0000 7fea10d46700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea0c0f5410 0x7fea0c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:34554/0 (socket says 172.21.15.148:34554) 2023-12-21T03:42:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.248+0000 7fea10d46700 1 -- 172.21.15.148:0/1120199670 learned_addr learned my addr 172.21.15.148:0/1120199670 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.248+0000 7fea10d46700 1 -- 172.21.15.148:0/1120199670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea00005040 con 0x7fea0c0f5410 2023-12-21T03:42:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.249+0000 7fea10d46700 1 --2- 172.21.15.148:0/1120199670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea0c0f5410 0x7fea0c105630 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fea00013510 tx=0x7fea0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.249+0000 7fea097fa700 1 -- 172.21.15.148:0/1120199670 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea00009320 con 0x7fea0c0f5410 2023-12-21T03:42:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.249+0000 7fea12faa700 1 -- 172.21.15.148:0/1120199670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea0c105d20 con 0x7fea0c0f5410 2023-12-21T03:42:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.249+0000 7fea097fa700 1 -- 172.21.15.148:0/1120199670 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea00013f00 con 0x7fea0c0f5410 2023-12-21T03:42:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.249+0000 7fea097fa700 1 -- 172.21.15.148:0/1120199670 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea00014d90 con 0x7fea0c0f5410 2023-12-21T03:42:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.249+0000 7fea12faa700 1 -- 172.21.15.148:0/1120199670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea0c1061b0 con 0x7fea0c0f5410 2023-12-21T03:42:04.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.249+0000 7fea097fa700 1 -- 172.21.15.148:0/1120199670 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea000095f0 con 0x7fea0c0f5410 2023-12-21T03:42:04.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.250+0000 7fea12faa700 1 -- 172.21.15.148:0/1120199670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0c045ad0 con 0x7fea0c0f5410 2023-12-21T03:42:04.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.250+0000 7fea097fa700 1 --2- 172.21.15.148:0/1120199670 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe9fc02ff60 0x7fe9fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:04.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.250+0000 7fea097fa700 1 -- 172.21.15.148:0/1120199670 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea00041000 con 0x7fea0c0f5410 2023-12-21T03:42:04.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.250+0000 7fea0bfff700 1 --2- 172.21.15.148:0/1120199670 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe9fc02ff60 0x7fe9fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:04.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.250+0000 7fea0bfff700 1 --2- 172.21.15.148:0/1120199670 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe9fc02ff60 0x7fe9fc032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fe9f80096a0 tx=0x7fe9f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:04.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.255+0000 7fea097fa700 1 -- 172.21.15.148:0/1120199670 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea00014420 con 0x7fea0c0f5410 2023-12-21T03:42:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.460+0000 7fea12faa700 1 -- 172.21.15.148:0/1120199670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea0c0fae60 con 0x7fea0c0f5410 2023-12-21T03:42:04.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.461+0000 7fea097fa700 1 -- 172.21.15.148:0/1120199670 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea00010230 con 0x7fea0c0f5410 2023-12-21T03:42:04.462 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:04.462 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.463+0000 7fea12faa700 1 -- 172.21.15.148:0/1120199670 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe9fc02ff60 msgr2=0x7fe9fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.463+0000 7fea12faa700 1 --2- 172.21.15.148:0/1120199670 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe9fc02ff60 0x7fe9fc032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fe9f80096a0 tx=0x7fe9f8006b40).stop 2023-12-21T03:42:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.463+0000 7fea12faa700 1 -- 172.21.15.148:0/1120199670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea0c0f5410 msgr2=0x7fea0c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.463+0000 7fea12faa700 1 --2- 172.21.15.148:0/1120199670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea0c0f5410 0x7fea0c105630 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fea00013510 tx=0x7fea0000b040).stop 2023-12-21T03:42:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.463+0000 7fea12faa700 1 -- 172.21.15.148:0/1120199670 shutdown_connections 2023-12-21T03:42:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.463+0000 7fea12faa700 1 --2- 172.21.15.148:0/1120199670 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe9fc02ff60 0x7fe9fc032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.463+0000 7fea12faa700 1 --2- 172.21.15.148:0/1120199670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea0c0f5410 0x7fea0c105630 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.463+0000 7fea12faa700 1 -- 172.21.15.148:0/1120199670 >> 172.21.15.148:0/1120199670 conn(0x7fea0c0f0b70 msgr2=0x7fea0c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.463+0000 7fea12faa700 1 -- 172.21.15.148:0/1120199670 shutdown_connections 2023-12-21T03:42:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:04.463+0000 7fea12faa700 1 -- 172.21.15.148:0/1120199670 wait complete. 2023-12-21T03:42:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:05.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:04.425338+0000 mgr.smithi078.zidbgq (mgr.14162) 82 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:05.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:04.461548+0000 mon.smithi078 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.148:0/1120199670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:05.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:05.848 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:06.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:06 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:06.425571+0000 mgr.smithi078.zidbgq (mgr.14162) 83 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:07.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.001+0000 7fe3a5acc700 1 -- 172.21.15.148:0/1234651242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3a0060180 msgr2=0x7fe3a00f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.001+0000 7fe3a5acc700 1 --2- 172.21.15.148:0/1234651242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3a0060180 0x7fe3a00f8f80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fe394009fb0 tx=0x7fe394009b30).stop 2023-12-21T03:42:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.001+0000 7fe3a5acc700 1 -- 172.21.15.148:0/1234651242 shutdown_connections 2023-12-21T03:42:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.001+0000 7fe3a5acc700 1 --2- 172.21.15.148:0/1234651242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3a0060180 0x7fe3a00f8f80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.001+0000 7fe3a5acc700 1 -- 172.21.15.148:0/1234651242 >> 172.21.15.148:0/1234651242 conn(0x7fe3a00f09a0 msgr2=0x7fe3a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.001+0000 7fe3a5acc700 1 -- 172.21.15.148:0/1234651242 shutdown_connections 2023-12-21T03:42:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.001+0000 7fe3a5acc700 1 -- 172.21.15.148:0/1234651242 wait complete. 2023-12-21T03:42:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.001+0000 7fe3a5acc700 1 Processor -- start 2023-12-21T03:42:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.001+0000 7fe3a5acc700 1 -- start start 2023-12-21T03:42:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.002+0000 7fe3a5acc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3a0060180 0x7fe3a0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.002+0000 7fe3a5acc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a01069f0 con 0x7fe3a0060180 2023-12-21T03:42:07.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.002+0000 7fe3a4aca700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3a0060180 0x7fe3a0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:07.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.002+0000 7fe3a4aca700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3a0060180 0x7fe3a0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:34574/0 (socket says 172.21.15.148:34574) 2023-12-21T03:42:07.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.002+0000 7fe3a4aca700 1 -- 172.21.15.148:0/2515219113 learned_addr learned my addr 172.21.15.148:0/2515219113 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:07.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.002+0000 7fe3a4aca700 1 -- 172.21.15.148:0/2515219113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe394005040 con 0x7fe3a0060180 2023-12-21T03:42:07.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.002+0000 7fe3a4aca700 1 --2- 172.21.15.148:0/2515219113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3a0060180 0x7fe3a0106500 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fe394014000 tx=0x7fe39400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:07.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.003+0000 7fe39d7fa700 1 -- 172.21.15.148:0/2515219113 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3940095f0 con 0x7fe3a0060180 2023-12-21T03:42:07.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.003+0000 7fe3a5acc700 1 -- 172.21.15.148:0/2515219113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a0106bf0 con 0x7fe3a0060180 2023-12-21T03:42:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.003+0000 7fe39d7fa700 1 -- 172.21.15.148:0/2515219113 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe394007e70 con 0x7fe3a0060180 2023-12-21T03:42:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.003+0000 7fe39d7fa700 1 -- 172.21.15.148:0/2515219113 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe394014ea0 con 0x7fe3a0060180 2023-12-21T03:42:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.003+0000 7fe3a5acc700 1 -- 172.21.15.148:0/2515219113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a0107080 con 0x7fe3a0060180 2023-12-21T03:42:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.003+0000 7fe39d7fa700 1 -- 172.21.15.148:0/2515219113 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe394025070 con 0x7fe3a0060180 2023-12-21T03:42:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.003+0000 7fe3a5acc700 1 -- 172.21.15.148:0/2515219113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a0045ad0 con 0x7fe3a0060180 2023-12-21T03:42:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.004+0000 7fe39d7fa700 1 --2- 172.21.15.148:0/2515219113 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe39002ff60 0x7fe390032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.004+0000 7fe39d7fa700 1 -- 172.21.15.148:0/2515219113 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe394040a30 con 0x7fe3a0060180 2023-12-21T03:42:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.004+0000 7fe39ffff700 1 --2- 172.21.15.148:0/2515219113 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe39002ff60 0x7fe390032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.004+0000 7fe39ffff700 1 --2- 172.21.15.148:0/2515219113 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe39002ff60 0x7fe390032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fe38c0096a0 tx=0x7fe38c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:07.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.008+0000 7fe39d7fa700 1 -- 172.21.15.148:0/2515219113 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3940079c0 con 0x7fe3a0060180 2023-12-21T03:42:07.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.215+0000 7fe3a5acc700 1 -- 172.21.15.148:0/2515219113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3a0107f30 con 0x7fe3a0060180 2023-12-21T03:42:07.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.216+0000 7fe39d7fa700 1 -- 172.21.15.148:0/2515219113 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe39400f300 con 0x7fe3a0060180 2023-12-21T03:42:07.217 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:07.217 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:07.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.217+0000 7fe3a5acc700 1 -- 172.21.15.148:0/2515219113 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe39002ff60 msgr2=0x7fe390032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:07.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.217+0000 7fe3a5acc700 1 --2- 172.21.15.148:0/2515219113 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe39002ff60 0x7fe390032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fe38c0096a0 tx=0x7fe38c006b40).stop 2023-12-21T03:42:07.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.217+0000 7fe3a5acc700 1 -- 172.21.15.148:0/2515219113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3a0060180 msgr2=0x7fe3a0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:07.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.217+0000 7fe3a5acc700 1 --2- 172.21.15.148:0/2515219113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3a0060180 0x7fe3a0106500 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fe394014000 tx=0x7fe39400b040).stop 2023-12-21T03:42:07.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.218+0000 7fe3a5acc700 1 -- 172.21.15.148:0/2515219113 shutdown_connections 2023-12-21T03:42:07.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.218+0000 7fe3a5acc700 1 --2- 172.21.15.148:0/2515219113 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe39002ff60 0x7fe390032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:07.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.218+0000 7fe3a5acc700 1 --2- 172.21.15.148:0/2515219113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3a0060180 0x7fe3a0106500 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:07.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.218+0000 7fe3a5acc700 1 -- 172.21.15.148:0/2515219113 >> 172.21.15.148:0/2515219113 conn(0x7fe3a00f09a0 msgr2=0x7fe3a00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:07.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.218+0000 7fe3a5acc700 1 -- 172.21.15.148:0/2515219113 shutdown_connections 2023-12-21T03:42:07.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:07.218+0000 7fe3a5acc700 1 -- 172.21.15.148:0/2515219113 wait complete. 2023-12-21T03:42:07.220 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:07.216654+0000 mon.smithi078 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.148:0/2515219113' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:08.598 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:08.598 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:08.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:42:07.853772+0000 mgr.smithi078.zidbgq (mgr.14162) 84 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi078 ... 2023-12-21T03:42:08.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Verifying port 9100 ... 2023-12-21T03:42:08.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 1 from systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:42:08.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2023-12-21T03:42:08.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2023-12-21T03:42:08.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:42:08.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ports=daemon_ports) 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:42:08.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: Failed command: systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: yield (conn, connr) 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: code, '\n'.join(err))) 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi078 ... 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Verifying port 9100 ... 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 1 from systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:42:08.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ports=daemon_ports) 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: Failed command: systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:42:07.859265+0000 mgr.smithi078.zidbgq (mgr.14162) 85 : cephadm [INF] Deploying daemon prometheus.smithi078 on smithi078 2023-12-21T03:42:08.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:08.425887+0000 mgr.smithi078.zidbgq (mgr.14162) 86 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:09.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.665+0000 7fac173b0700 1 -- 172.21.15.148:0/3709059020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f57b0 msgr2=0x7fac100f5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:09.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.665+0000 7fac173b0700 1 --2- 172.21.15.148:0/3709059020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f57b0 0x7fac100f5bd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fac0c009fb0 tx=0x7fac0c009b30).stop 2023-12-21T03:42:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.665+0000 7fac173b0700 1 -- 172.21.15.148:0/3709059020 shutdown_connections 2023-12-21T03:42:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.665+0000 7fac173b0700 1 --2- 172.21.15.148:0/3709059020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f57b0 0x7fac100f5bd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.665+0000 7fac173b0700 1 -- 172.21.15.148:0/3709059020 >> 172.21.15.148:0/3709059020 conn(0x7fac100f0120 msgr2=0x7fac100f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.665+0000 7fac173b0700 1 -- 172.21.15.148:0/3709059020 shutdown_connections 2023-12-21T03:42:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.665+0000 7fac173b0700 1 -- 172.21.15.148:0/3709059020 wait complete. 2023-12-21T03:42:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.665+0000 7fac173b0700 1 Processor -- start 2023-12-21T03:42:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.665+0000 7fac173b0700 1 -- start start 2023-12-21T03:42:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.665+0000 7fac173b0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f57b0 0x7fac10109180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.665+0000 7fac173b0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac100f51d0 con 0x7fac100f57b0 2023-12-21T03:42:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.666+0000 7fac163ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f57b0 0x7fac10109180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.666+0000 7fac163ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f57b0 0x7fac10109180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:57652/0 (socket says 172.21.15.148:57652) 2023-12-21T03:42:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.666+0000 7fac163ae700 1 -- 172.21.15.148:0/625864720 learned_addr learned my addr 172.21.15.148:0/625864720 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.666+0000 7fac163ae700 1 -- 172.21.15.148:0/625864720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c005040 con 0x7fac100f57b0 2023-12-21T03:42:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.666+0000 7fac163ae700 1 --2- 172.21.15.148:0/625864720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f57b0 0x7fac10109180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fac0c013510 tx=0x7fac0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.666+0000 7fac06ffd700 1 -- 172.21.15.148:0/625864720 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac0c009320 con 0x7fac100f57b0 2023-12-21T03:42:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.666+0000 7fac173b0700 1 -- 172.21.15.148:0/625864720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac100f53d0 con 0x7fac100f57b0 2023-12-21T03:42:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.666+0000 7fac06ffd700 1 -- 172.21.15.148:0/625864720 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac0c013f00 con 0x7fac100f57b0 2023-12-21T03:42:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.666+0000 7fac06ffd700 1 -- 172.21.15.148:0/625864720 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac0c014dc0 con 0x7fac100f57b0 2023-12-21T03:42:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.666+0000 7fac173b0700 1 -- 172.21.15.148:0/625864720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac101098b0 con 0x7fac100f57b0 2023-12-21T03:42:09.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.667+0000 7fac06ffd700 1 -- 172.21.15.148:0/625864720 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac0c0095f0 con 0x7fac100f57b0 2023-12-21T03:42:09.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.667+0000 7fac06ffd700 1 --2- 172.21.15.148:0/625864720 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fabfc02ff60 0x7fabfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:09.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.667+0000 7fac173b0700 1 -- 172.21.15.148:0/625864720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac10045ad0 con 0x7fac100f57b0 2023-12-21T03:42:09.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.667+0000 7fac06ffd700 1 -- 172.21.15.148:0/625864720 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac0c0410c0 con 0x7fac100f57b0 2023-12-21T03:42:09.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.667+0000 7fac15bad700 1 --2- 172.21.15.148:0/625864720 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fabfc02ff60 0x7fabfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:09.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.668+0000 7fac15bad700 1 --2- 172.21.15.148:0/625864720 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fabfc02ff60 0x7fabfc032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fac000096a0 tx=0x7fac00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:09.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.672+0000 7fac06ffd700 1 -- 172.21.15.148:0/625864720 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac0c06d2b0 con 0x7fac100f57b0 2023-12-21T03:42:09.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.878+0000 7fac173b0700 1 -- 172.21.15.148:0/625864720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac100fabd0 con 0x7fac100f57b0 2023-12-21T03:42:09.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.879+0000 7fac06ffd700 1 -- 172.21.15.148:0/625864720 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac0c00f340 con 0x7fac100f57b0 2023-12-21T03:42:09.880 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:09.880 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:09.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.880+0000 7fac173b0700 1 -- 172.21.15.148:0/625864720 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fabfc02ff60 msgr2=0x7fabfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:09.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.880+0000 7fac173b0700 1 --2- 172.21.15.148:0/625864720 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fabfc02ff60 0x7fabfc032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fac000096a0 tx=0x7fac00006b40).stop 2023-12-21T03:42:09.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.880+0000 7fac173b0700 1 -- 172.21.15.148:0/625864720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f57b0 msgr2=0x7fac10109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:09.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.880+0000 7fac173b0700 1 --2- 172.21.15.148:0/625864720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f57b0 0x7fac10109180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fac0c013510 tx=0x7fac0c00b040).stop 2023-12-21T03:42:09.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.880+0000 7fac173b0700 1 -- 172.21.15.148:0/625864720 shutdown_connections 2023-12-21T03:42:09.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.880+0000 7fac173b0700 1 --2- 172.21.15.148:0/625864720 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fabfc02ff60 0x7fabfc032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:09.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.880+0000 7fac173b0700 1 --2- 172.21.15.148:0/625864720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f57b0 0x7fac10109180 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:09.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.880+0000 7fac173b0700 1 -- 172.21.15.148:0/625864720 >> 172.21.15.148:0/625864720 conn(0x7fac100f0120 msgr2=0x7fac100f0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:09.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.880+0000 7fac173b0700 1 -- 172.21.15.148:0/625864720 shutdown_connections 2023-12-21T03:42:09.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:09.880+0000 7fac173b0700 1 -- 172.21.15.148:0/625864720 wait complete. 2023-12-21T03:42:09.883 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:10.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:10 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:09.879519+0000 mon.smithi078 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.148:0/625864720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:11.284 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:11.285 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:11.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:10.426353+0000 mgr.smithi078.zidbgq (mgr.14162) 87 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.425+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/21057495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e080f5410 msgr2=0x7f1e080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.425+0000 7f1e0e2a9700 1 --2- 172.21.15.148:0/21057495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e080f5410 0x7f1e080f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1df8009fb0 tx=0x7f1df8009b30).stop 2023-12-21T03:42:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.425+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/21057495 shutdown_connections 2023-12-21T03:42:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.425+0000 7f1e0e2a9700 1 --2- 172.21.15.148:0/21057495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e080f5410 0x7f1e080f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.425+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/21057495 >> 172.21.15.148:0/21057495 conn(0x7f1e080f0b70 msgr2=0x7f1e080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.425+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/21057495 shutdown_connections 2023-12-21T03:42:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.425+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/21057495 wait complete. 2023-12-21T03:42:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.426+0000 7f1e0e2a9700 1 Processor -- start 2023-12-21T03:42:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.426+0000 7f1e0e2a9700 1 -- start start 2023-12-21T03:42:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.426+0000 7f1e0e2a9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.426+0000 7f1e0e2a9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e08060240 con 0x7f1e080f5410 2023-12-21T03:42:12.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.426+0000 7f1e07fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:12.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.426+0000 7f1e07fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:57674/0 (socket says 172.21.15.148:57674) 2023-12-21T03:42:12.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.426+0000 7f1e07fff700 1 -- 172.21.15.148:0/3732416958 learned_addr learned my addr 172.21.15.148:0/3732416958 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:12.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.426+0000 7f1e07fff700 1 -- 172.21.15.148:0/3732416958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df8005040 con 0x7f1e080f5410 2023-12-21T03:42:12.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.427+0000 7f1e07fff700 1 --2- 172.21.15.148:0/3732416958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f1df8013510 tx=0x7f1df800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:12.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.427+0000 7f1e04ff9700 1 -- 172.21.15.148:0/3732416958 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df8009320 con 0x7f1e080f5410 2023-12-21T03:42:12.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.427+0000 7f1e04ff9700 1 -- 172.21.15.148:0/3732416958 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1df8013f00 con 0x7f1e080f5410 2023-12-21T03:42:12.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.427+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/3732416958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e08060440 con 0x7f1e080f5410 2023-12-21T03:42:12.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.427+0000 7f1e04ff9700 1 -- 172.21.15.148:0/3732416958 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df8014d90 con 0x7f1e080f5410 2023-12-21T03:42:12.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.427+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/3732416958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e08109ae0 con 0x7f1e080f5410 2023-12-21T03:42:12.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.428+0000 7f1e04ff9700 1 -- 172.21.15.148:0/3732416958 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1df8014420 con 0x7f1e080f5410 2023-12-21T03:42:12.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.428+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/3732416958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e08045ad0 con 0x7f1e080f5410 2023-12-21T03:42:12.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.428+0000 7f1e04ff9700 1 --2- 172.21.15.148:0/3732416958 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1df002ff60 0x7f1df0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:12.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.428+0000 7f1e04ff9700 1 -- 172.21.15.148:0/3732416958 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1df8041000 con 0x7f1e080f5410 2023-12-21T03:42:12.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.428+0000 7f1e077fe700 1 --2- 172.21.15.148:0/3732416958 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1df002ff60 0x7f1df0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:12.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.428+0000 7f1e077fe700 1 --2- 172.21.15.148:0/3732416958 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1df002ff60 0x7f1df0032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f1dfc0096a0 tx=0x7f1dfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:12.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.432+0000 7f1e04ff9700 1 -- 172.21.15.148:0/3732416958 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1df80095f0 con 0x7f1e080f5410 2023-12-21T03:42:12.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.638+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/3732416958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e080fae60 con 0x7f1e080f5410 2023-12-21T03:42:12.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.638+0000 7f1e04ff9700 1 -- 172.21.15.148:0/3732416958 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1df801db70 con 0x7f1e080f5410 2023-12-21T03:42:12.640 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:12.640 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.640+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/3732416958 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1df002ff60 msgr2=0x7f1df0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.640+0000 7f1e0e2a9700 1 --2- 172.21.15.148:0/3732416958 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1df002ff60 0x7f1df0032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f1dfc0096a0 tx=0x7f1dfc006b40).stop 2023-12-21T03:42:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.640+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/3732416958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e080f5410 msgr2=0x7f1e08109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.640+0000 7f1e0e2a9700 1 --2- 172.21.15.148:0/3732416958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f1df8013510 tx=0x7f1df800b040).stop 2023-12-21T03:42:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.640+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/3732416958 shutdown_connections 2023-12-21T03:42:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.640+0000 7f1e0e2a9700 1 --2- 172.21.15.148:0/3732416958 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1df002ff60 0x7f1df0032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.640+0000 7f1e0e2a9700 1 --2- 172.21.15.148:0/3732416958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.640+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/3732416958 >> 172.21.15.148:0/3732416958 conn(0x7f1e080f0b70 msgr2=0x7f1e080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:12.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.640+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/3732416958 shutdown_connections 2023-12-21T03:42:12.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:12.640+0000 7f1e0e2a9700 1 -- 172.21.15.148:0/3732416958 wait complete. 2023-12-21T03:42:12.642 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:12.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:12.426597+0000 mgr.smithi078.zidbgq (mgr.14162) 88 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:12.639201+0000 mon.smithi078 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.148:0/3732416958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:14.009 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:14.009 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:15.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.108+0000 7f6323063700 1 -- 172.21.15.148:0/2245325919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631c0f5410 msgr2=0x7f631c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:15.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.108+0000 7f6323063700 1 --2- 172.21.15.148:0/2245325919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631c0f5410 0x7f631c0f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f6310009fb0 tx=0x7f6310009b30).stop 2023-12-21T03:42:15.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.108+0000 7f6323063700 1 -- 172.21.15.148:0/2245325919 shutdown_connections 2023-12-21T03:42:15.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.108+0000 7f6323063700 1 --2- 172.21.15.148:0/2245325919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631c0f5410 0x7f631c0f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.108+0000 7f6323063700 1 -- 172.21.15.148:0/2245325919 >> 172.21.15.148:0/2245325919 conn(0x7f631c0f0b70 msgr2=0x7f631c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.108+0000 7f6323063700 1 -- 172.21.15.148:0/2245325919 shutdown_connections 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.108+0000 7f6323063700 1 -- 172.21.15.148:0/2245325919 wait complete. 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.108+0000 7f6323063700 1 Processor -- start 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.108+0000 7f6323063700 1 -- start start 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.109+0000 7f6323063700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631c0f5410 0x7f631c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.109+0000 7f6323063700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631c105b20 con 0x7f631c0f5410 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.109+0000 7f6320dff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631c0f5410 0x7f631c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.109+0000 7f6320dff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631c0f5410 0x7f631c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:57698/0 (socket says 172.21.15.148:57698) 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.109+0000 7f6320dff700 1 -- 172.21.15.148:0/1890600726 learned_addr learned my addr 172.21.15.148:0/1890600726 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.109+0000 7f6320dff700 1 -- 172.21.15.148:0/1890600726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6310005040 con 0x7f631c0f5410 2023-12-21T03:42:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.109+0000 7f6320dff700 1 --2- 172.21.15.148:0/1890600726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631c0f5410 0x7f631c105630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f6310013510 tx=0x7f631000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:15.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.109+0000 7f63197fa700 1 -- 172.21.15.148:0/1890600726 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6310009320 con 0x7f631c0f5410 2023-12-21T03:42:15.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.110+0000 7f6323063700 1 -- 172.21.15.148:0/1890600726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631c105d20 con 0x7f631c0f5410 2023-12-21T03:42:15.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.110+0000 7f63197fa700 1 -- 172.21.15.148:0/1890600726 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6310013f00 con 0x7f631c0f5410 2023-12-21T03:42:15.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.110+0000 7f63197fa700 1 -- 172.21.15.148:0/1890600726 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6310014d90 con 0x7f631c0f5410 2023-12-21T03:42:15.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.110+0000 7f6323063700 1 -- 172.21.15.148:0/1890600726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631c1061b0 con 0x7f631c0f5410 2023-12-21T03:42:15.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.110+0000 7f63197fa700 1 -- 172.21.15.148:0/1890600726 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63100095f0 con 0x7f631c0f5410 2023-12-21T03:42:15.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.110+0000 7f6323063700 1 -- 172.21.15.148:0/1890600726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631c045ad0 con 0x7f631c0f5410 2023-12-21T03:42:15.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.111+0000 7f63197fa700 1 --2- 172.21.15.148:0/1890600726 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f630c02ff60 0x7f630c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:15.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.111+0000 7f63197fa700 1 -- 172.21.15.148:0/1890600726 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6310041000 con 0x7f631c0f5410 2023-12-21T03:42:15.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.111+0000 7f631bfff700 1 --2- 172.21.15.148:0/1890600726 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f630c02ff60 0x7f630c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:15.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.111+0000 7f631bfff700 1 --2- 172.21.15.148:0/1890600726 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f630c02ff60 0x7f630c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f63080096a0 tx=0x7f6308006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:15.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.115+0000 7f63197fa700 1 -- 172.21.15.148:0/1890600726 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f631001d7f0 con 0x7f631c0f5410 2023-12-21T03:42:15.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.320+0000 7f6323063700 1 -- 172.21.15.148:0/1890600726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f631c0fae60 con 0x7f631c0f5410 2023-12-21T03:42:15.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.321+0000 7f63197fa700 1 -- 172.21.15.148:0/1890600726 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6310010230 con 0x7f631c0f5410 2023-12-21T03:42:15.322 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:15.322 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:15.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.323+0000 7f6323063700 1 -- 172.21.15.148:0/1890600726 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f630c02ff60 msgr2=0x7f630c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:15.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.323+0000 7f6323063700 1 --2- 172.21.15.148:0/1890600726 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f630c02ff60 0x7f630c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f63080096a0 tx=0x7f6308006b40).stop 2023-12-21T03:42:15.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.323+0000 7f6323063700 1 -- 172.21.15.148:0/1890600726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631c0f5410 msgr2=0x7f631c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:15.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.323+0000 7f6323063700 1 --2- 172.21.15.148:0/1890600726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631c0f5410 0x7f631c105630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f6310013510 tx=0x7f631000b040).stop 2023-12-21T03:42:15.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.323+0000 7f6323063700 1 -- 172.21.15.148:0/1890600726 shutdown_connections 2023-12-21T03:42:15.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.323+0000 7f6323063700 1 --2- 172.21.15.148:0/1890600726 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f630c02ff60 0x7f630c032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:15.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.323+0000 7f6323063700 1 --2- 172.21.15.148:0/1890600726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631c0f5410 0x7f631c105630 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:15.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.323+0000 7f6323063700 1 -- 172.21.15.148:0/1890600726 >> 172.21.15.148:0/1890600726 conn(0x7f631c0f0b70 msgr2=0x7f631c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:15.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.323+0000 7f6323063700 1 -- 172.21.15.148:0/1890600726 shutdown_connections 2023-12-21T03:42:15.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:15.323+0000 7f6323063700 1 -- 172.21.15.148:0/1890600726 wait complete. 2023-12-21T03:42:15.325 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:15.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:14.426944+0000 mgr.smithi078.zidbgq (mgr.14162) 89 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:16.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:16 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:15.322013+0000 mon.smithi078 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.148:0/1890600726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:16.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:16.699 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:17.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:16.427251+0000 mgr.smithi078.zidbgq (mgr.14162) 90 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:17.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.843+0000 7fe65adc1700 1 -- 172.21.15.148:0/84085283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6540f4740 msgr2=0x7fe6540f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:17.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.843+0000 7fe65adc1700 1 --2- 172.21.15.148:0/84085283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6540f4740 0x7fe6540f4b60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fe648009fb0 tx=0x7fe648009b30).stop 2023-12-21T03:42:17.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.843+0000 7fe65adc1700 1 -- 172.21.15.148:0/84085283 shutdown_connections 2023-12-21T03:42:17.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.843+0000 7fe65adc1700 1 --2- 172.21.15.148:0/84085283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6540f4740 0x7fe6540f4b60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:17.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.843+0000 7fe65adc1700 1 -- 172.21.15.148:0/84085283 >> 172.21.15.148:0/84085283 conn(0x7fe6540f0260 msgr2=0x7fe6540f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:17.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.843+0000 7fe65adc1700 1 -- 172.21.15.148:0/84085283 shutdown_connections 2023-12-21T03:42:17.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.843+0000 7fe65adc1700 1 -- 172.21.15.148:0/84085283 wait complete. 2023-12-21T03:42:17.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.844+0000 7fe65adc1700 1 Processor -- start 2023-12-21T03:42:17.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.844+0000 7fe65adc1700 1 -- start start 2023-12-21T03:42:17.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.844+0000 7fe65adc1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6540f4740 0x7fe6541066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:17.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.844+0000 7fe65adc1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe654106bc0 con 0x7fe6540f4740 2023-12-21T03:42:17.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.845+0000 7fe658b5d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6540f4740 0x7fe6541066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:17.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.845+0000 7fe658b5d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6540f4740 0x7fe6541066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:57704/0 (socket says 172.21.15.148:57704) 2023-12-21T03:42:17.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.845+0000 7fe658b5d700 1 -- 172.21.15.148:0/1138609496 learned_addr learned my addr 172.21.15.148:0/1138609496 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.845+0000 7fe658b5d700 1 -- 172.21.15.148:0/1138609496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe648005040 con 0x7fe6540f4740 2023-12-21T03:42:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.845+0000 7fe658b5d700 1 --2- 172.21.15.148:0/1138609496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6540f4740 0x7fe6541066d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe648013510 tx=0x7fe64800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.845+0000 7fe6517fa700 1 -- 172.21.15.148:0/1138609496 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe648009320 con 0x7fe6540f4740 2023-12-21T03:42:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.845+0000 7fe65adc1700 1 -- 172.21.15.148:0/1138609496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe654106dc0 con 0x7fe6540f4740 2023-12-21T03:42:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.845+0000 7fe6517fa700 1 -- 172.21.15.148:0/1138609496 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe648013f00 con 0x7fe6540f4740 2023-12-21T03:42:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.845+0000 7fe6517fa700 1 -- 172.21.15.148:0/1138609496 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe648014d90 con 0x7fe6540f4740 2023-12-21T03:42:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.845+0000 7fe65adc1700 1 -- 172.21.15.148:0/1138609496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe654107250 con 0x7fe6540f4740 2023-12-21T03:42:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.846+0000 7fe6517fa700 1 -- 172.21.15.148:0/1138609496 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe648014420 con 0x7fe6540f4740 2023-12-21T03:42:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.846+0000 7fe65adc1700 1 -- 172.21.15.148:0/1138609496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe654045ad0 con 0x7fe6540f4740 2023-12-21T03:42:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.846+0000 7fe6517fa700 1 --2- 172.21.15.148:0/1138609496 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe64402ff60 0x7fe644032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.846+0000 7fe6517fa700 1 -- 172.21.15.148:0/1138609496 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe6480410c0 con 0x7fe6540f4740 2023-12-21T03:42:17.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.847+0000 7fe653fff700 1 --2- 172.21.15.148:0/1138609496 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe64402ff60 0x7fe644032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:17.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.847+0000 7fe653fff700 1 --2- 172.21.15.148:0/1138609496 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe64402ff60 0x7fe644032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe6400096a0 tx=0x7fe640006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:17.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:17.851+0000 7fe6517fa700 1 -- 172.21.15.148:0/1138609496 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe6480095f0 con 0x7fe6540f4740 2023-12-21T03:42:18.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.058+0000 7fe65adc1700 1 -- 172.21.15.148:0/1138609496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe654108100 con 0x7fe6540f4740 2023-12-21T03:42:18.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.059+0000 7fe6517fa700 1 -- 172.21.15.148:0/1138609496 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe64801db70 con 0x7fe6540f4740 2023-12-21T03:42:18.060 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:18.060 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:18.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.060+0000 7fe65adc1700 1 -- 172.21.15.148:0/1138609496 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe64402ff60 msgr2=0x7fe644032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:18.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.060+0000 7fe65adc1700 1 --2- 172.21.15.148:0/1138609496 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe64402ff60 0x7fe644032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe6400096a0 tx=0x7fe640006b40).stop 2023-12-21T03:42:18.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.060+0000 7fe65adc1700 1 -- 172.21.15.148:0/1138609496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6540f4740 msgr2=0x7fe6541066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:18.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.060+0000 7fe65adc1700 1 --2- 172.21.15.148:0/1138609496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6540f4740 0x7fe6541066d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe648013510 tx=0x7fe64800b040).stop 2023-12-21T03:42:18.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.061+0000 7fe65adc1700 1 -- 172.21.15.148:0/1138609496 shutdown_connections 2023-12-21T03:42:18.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.061+0000 7fe65adc1700 1 --2- 172.21.15.148:0/1138609496 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe64402ff60 0x7fe644032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:18.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.061+0000 7fe65adc1700 1 --2- 172.21.15.148:0/1138609496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6540f4740 0x7fe6541066d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:18.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.061+0000 7fe65adc1700 1 -- 172.21.15.148:0/1138609496 >> 172.21.15.148:0/1138609496 conn(0x7fe6540f0260 msgr2=0x7fe6540f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:18.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.061+0000 7fe65adc1700 1 -- 172.21.15.148:0/1138609496 shutdown_connections 2023-12-21T03:42:18.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:18.061+0000 7fe65adc1700 1 -- 172.21.15.148:0/1138609496 wait complete. 2023-12-21T03:42:18.063 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:18.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:18 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:18.059765+0000 mon.smithi078 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.148:0/1138609496' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:19.459 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:19.460 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:19.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:18.427538+0000 mgr.smithi078.zidbgq (mgr.14162) 91 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:20.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.579+0000 7f5efe321700 1 -- 172.21.15.148:0/2081609091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ef80f62a0 msgr2=0x7f5ef80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:20.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.579+0000 7f5efe321700 1 --2- 172.21.15.148:0/2081609091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ef80f62a0 0x7f5ef80f66c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f5ef4009fb0 tx=0x7f5ef4009b30).stop 2023-12-21T03:42:20.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.579+0000 7f5efe321700 1 -- 172.21.15.148:0/2081609091 shutdown_connections 2023-12-21T03:42:20.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.579+0000 7f5efe321700 1 --2- 172.21.15.148:0/2081609091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ef80f62a0 0x7f5ef80f66c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:20.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.579+0000 7f5efe321700 1 -- 172.21.15.148:0/2081609091 >> 172.21.15.148:0/2081609091 conn(0x7f5ef80f1aa0 msgr2=0x7f5ef80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.579+0000 7f5efe321700 1 -- 172.21.15.148:0/2081609091 shutdown_connections 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.579+0000 7f5efe321700 1 -- 172.21.15.148:0/2081609091 wait complete. 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.579+0000 7f5efe321700 1 Processor -- start 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.579+0000 7f5efe321700 1 -- start start 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.579+0000 7f5efe321700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ef80f62a0 0x7f5ef81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.579+0000 7f5efe321700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81069a0 con 0x7f5ef80f62a0 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.580+0000 7f5efd31f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ef80f62a0 0x7f5ef81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.580+0000 7f5efd31f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ef80f62a0 0x7f5ef81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:50866/0 (socket says 172.21.15.148:50866) 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.580+0000 7f5efd31f700 1 -- 172.21.15.148:0/715352623 learned_addr learned my addr 172.21.15.148:0/715352623 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.580+0000 7f5efd31f700 1 -- 172.21.15.148:0/715352623 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef4005040 con 0x7f5ef80f62a0 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.580+0000 7f5efd31f700 1 --2- 172.21.15.148:0/715352623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ef80f62a0 0x7f5ef81064b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5ef4009830 tx=0x7f5ef400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.580+0000 7f5eedffb700 1 -- 172.21.15.148:0/715352623 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ef4009320 con 0x7f5ef80f62a0 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.580+0000 7f5eedffb700 1 -- 172.21.15.148:0/715352623 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ef40136c0 con 0x7f5ef80f62a0 2023-12-21T03:42:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.580+0000 7f5efe321700 1 -- 172.21.15.148:0/715352623 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef8106ba0 con 0x7f5ef80f62a0 2023-12-21T03:42:20.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.580+0000 7f5eedffb700 1 -- 172.21.15.148:0/715352623 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ef401ee90 con 0x7f5ef80f62a0 2023-12-21T03:42:20.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.580+0000 7f5efe321700 1 -- 172.21.15.148:0/715352623 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef8107030 con 0x7f5ef80f62a0 2023-12-21T03:42:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.581+0000 7f5eedffb700 1 -- 172.21.15.148:0/715352623 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ef401e420 con 0x7f5ef80f62a0 2023-12-21T03:42:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.581+0000 7f5efe321700 1 -- 172.21.15.148:0/715352623 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef8045ad0 con 0x7f5ef80f62a0 2023-12-21T03:42:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.581+0000 7f5eedffb700 1 --2- 172.21.15.148:0/715352623 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ee402ff10 0x7f5ee40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.581+0000 7f5eedffb700 1 -- 172.21.15.148:0/715352623 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ef4041050 con 0x7f5ef80f62a0 2023-12-21T03:42:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.582+0000 7f5efcb1e700 1 --2- 172.21.15.148:0/715352623 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ee402ff10 0x7f5ee40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.582+0000 7f5efcb1e700 1 --2- 172.21.15.148:0/715352623 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ee402ff10 0x7f5ee40323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5ee80096a0 tx=0x7f5ee8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.586+0000 7f5eedffb700 1 -- 172.21.15.148:0/715352623 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ef406d2b0 con 0x7f5ef80f62a0 2023-12-21T03:42:20.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.793+0000 7f5efe321700 1 -- 172.21.15.148:0/715352623 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ef8107ee0 con 0x7f5ef80f62a0 2023-12-21T03:42:20.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.794+0000 7f5eedffb700 1 -- 172.21.15.148:0/715352623 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ef40095f0 con 0x7f5ef80f62a0 2023-12-21T03:42:20.796 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:20.796 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:20.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.796+0000 7f5efe321700 1 -- 172.21.15.148:0/715352623 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ee402ff10 msgr2=0x7f5ee40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:20.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.796+0000 7f5efe321700 1 --2- 172.21.15.148:0/715352623 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ee402ff10 0x7f5ee40323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5ee80096a0 tx=0x7f5ee8006b40).stop 2023-12-21T03:42:20.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.796+0000 7f5efe321700 1 -- 172.21.15.148:0/715352623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ef80f62a0 msgr2=0x7f5ef81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:20.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.796+0000 7f5efe321700 1 --2- 172.21.15.148:0/715352623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ef80f62a0 0x7f5ef81064b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5ef4009830 tx=0x7f5ef400b040).stop 2023-12-21T03:42:20.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.796+0000 7f5efe321700 1 -- 172.21.15.148:0/715352623 shutdown_connections 2023-12-21T03:42:20.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.796+0000 7f5efe321700 1 --2- 172.21.15.148:0/715352623 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ee402ff10 0x7f5ee40323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:20.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.796+0000 7f5efe321700 1 --2- 172.21.15.148:0/715352623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ef80f62a0 0x7f5ef81064b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:20.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.796+0000 7f5efe321700 1 -- 172.21.15.148:0/715352623 >> 172.21.15.148:0/715352623 conn(0x7f5ef80f1aa0 msgr2=0x7f5ef80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:20.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.796+0000 7f5efe321700 1 -- 172.21.15.148:0/715352623 shutdown_connections 2023-12-21T03:42:20.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:20.796+0000 7f5efe321700 1 -- 172.21.15.148:0/715352623 wait complete. 2023-12-21T03:42:20.799 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:20.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:20 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:20.427892+0000 mgr.smithi078.zidbgq (mgr.14162) 92 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:20.795997+0000 mon.smithi078 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.148:0/715352623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:22.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:22.172 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:22.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:22.428279+0000 mgr.smithi078.zidbgq (mgr.14162) 93 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:23.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.273+0000 7fdec40e5700 1 -- 172.21.15.148:0/1834247068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdebc0f5410 msgr2=0x7fdebc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:23.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.273+0000 7fdec40e5700 1 --2- 172.21.15.148:0/1834247068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdebc0f5410 0x7fdebc0f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fdeac009fb0 tx=0x7fdeac009b30).stop 2023-12-21T03:42:23.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.273+0000 7fdec40e5700 1 -- 172.21.15.148:0/1834247068 shutdown_connections 2023-12-21T03:42:23.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.273+0000 7fdec40e5700 1 --2- 172.21.15.148:0/1834247068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdebc0f5410 0x7fdebc0f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:23.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.273+0000 7fdec40e5700 1 -- 172.21.15.148:0/1834247068 >> 172.21.15.148:0/1834247068 conn(0x7fdebc0f0b70 msgr2=0x7fdebc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:23.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.273+0000 7fdec40e5700 1 -- 172.21.15.148:0/1834247068 shutdown_connections 2023-12-21T03:42:23.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.273+0000 7fdec40e5700 1 -- 172.21.15.148:0/1834247068 wait complete. 2023-12-21T03:42:23.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.274+0000 7fdec40e5700 1 Processor -- start 2023-12-21T03:42:23.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.274+0000 7fdec40e5700 1 -- start start 2023-12-21T03:42:23.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.274+0000 7fdec40e5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdebc0601e0 0x7fdebc1095a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:23.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.274+0000 7fdec40e5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc109a90 con 0x7fdebc0601e0 2023-12-21T03:42:23.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.274+0000 7fdec1e81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdebc0601e0 0x7fdebc1095a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:23.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.274+0000 7fdec1e81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdebc0601e0 0x7fdebc1095a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:50894/0 (socket says 172.21.15.148:50894) 2023-12-21T03:42:23.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.274+0000 7fdec1e81700 1 -- 172.21.15.148:0/478804052 learned_addr learned my addr 172.21.15.148:0/478804052 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:23.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.275+0000 7fdec1e81700 1 -- 172.21.15.148:0/478804052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac005040 con 0x7fdebc0601e0 2023-12-21T03:42:23.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.275+0000 7fdec1e81700 1 --2- 172.21.15.148:0/478804052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdebc0601e0 0x7fdebc1095a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fdeac009f80 tx=0x7fdeac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:23.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.275+0000 7fdeb27fc700 1 -- 172.21.15.148:0/478804052 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeac0136c0 con 0x7fdebc0601e0 2023-12-21T03:42:23.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.275+0000 7fdeb27fc700 1 -- 172.21.15.148:0/478804052 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdeac0095f0 con 0x7fdebc0601e0 2023-12-21T03:42:23.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.275+0000 7fdec40e5700 1 -- 172.21.15.148:0/478804052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdebc109bd0 con 0x7fdebc0601e0 2023-12-21T03:42:23.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.275+0000 7fdeb27fc700 1 -- 172.21.15.148:0/478804052 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeac014420 con 0x7fdebc0601e0 2023-12-21T03:42:23.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.275+0000 7fdec40e5700 1 -- 172.21.15.148:0/478804052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdebc10a000 con 0x7fdebc0601e0 2023-12-21T03:42:23.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.276+0000 7fdec40e5700 1 -- 172.21.15.148:0/478804052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdebc045ad0 con 0x7fdebc0601e0 2023-12-21T03:42:23.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.276+0000 7fdeb27fc700 1 -- 172.21.15.148:0/478804052 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdeac025060 con 0x7fdebc0601e0 2023-12-21T03:42:23.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.276+0000 7fdeb27fc700 1 --2- 172.21.15.148:0/478804052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdea802ff60 0x7fdea8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:23.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.276+0000 7fdeb27fc700 1 -- 172.21.15.148:0/478804052 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdeac040e20 con 0x7fdebc0601e0 2023-12-21T03:42:23.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.276+0000 7fdec1680700 1 --2- 172.21.15.148:0/478804052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdea802ff60 0x7fdea8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:23.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.277+0000 7fdec1680700 1 --2- 172.21.15.148:0/478804052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdea802ff60 0x7fdea8032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fdeb80096a0 tx=0x7fdeb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:23.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.281+0000 7fdeb27fc700 1 -- 172.21.15.148:0/478804052 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdeac01cb60 con 0x7fdebc0601e0 2023-12-21T03:42:23.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.489+0000 7fdec40e5700 1 -- 172.21.15.148:0/478804052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdebc0fae60 con 0x7fdebc0601e0 2023-12-21T03:42:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.490+0000 7fdeb27fc700 1 -- 172.21.15.148:0/478804052 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdeac00f280 con 0x7fdebc0601e0 2023-12-21T03:42:23.492 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:23.492 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.491+0000 7fdec40e5700 1 -- 172.21.15.148:0/478804052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdea802ff60 msgr2=0x7fdea8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.491+0000 7fdec40e5700 1 --2- 172.21.15.148:0/478804052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdea802ff60 0x7fdea8032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fdeb80096a0 tx=0x7fdeb8006b40).stop 2023-12-21T03:42:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.491+0000 7fdec40e5700 1 -- 172.21.15.148:0/478804052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdebc0601e0 msgr2=0x7fdebc1095a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.491+0000 7fdec40e5700 1 --2- 172.21.15.148:0/478804052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdebc0601e0 0x7fdebc1095a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fdeac009f80 tx=0x7fdeac009b30).stop 2023-12-21T03:42:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.491+0000 7fdec40e5700 1 -- 172.21.15.148:0/478804052 shutdown_connections 2023-12-21T03:42:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.491+0000 7fdec40e5700 1 --2- 172.21.15.148:0/478804052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdea802ff60 0x7fdea8032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.491+0000 7fdec40e5700 1 --2- 172.21.15.148:0/478804052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdebc0601e0 0x7fdebc1095a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.491+0000 7fdec40e5700 1 -- 172.21.15.148:0/478804052 >> 172.21.15.148:0/478804052 conn(0x7fdebc0f0b70 msgr2=0x7fdebc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.491+0000 7fdec40e5700 1 -- 172.21.15.148:0/478804052 shutdown_connections 2023-12-21T03:42:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:23.491+0000 7fdec40e5700 1 -- 172.21.15.148:0/478804052 wait complete. 2023-12-21T03:42:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:23.491671+0000 mon.smithi078 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.148:0/478804052' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:24.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:24.889 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:25.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:24.428638+0000 mgr.smithi078.zidbgq (mgr.14162) 94 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.991+0000 7f5e57fff700 1 -- 172.21.15.148:0/4145650720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e580f62a0 msgr2=0x7f5e580f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.991+0000 7f5e57fff700 1 --2- 172.21.15.148:0/4145650720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e580f62a0 0x7f5e580f66c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5e48009fb0 tx=0x7f5e48009b30).stop 2023-12-21T03:42:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.991+0000 7f5e57fff700 1 -- 172.21.15.148:0/4145650720 shutdown_connections 2023-12-21T03:42:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.991+0000 7f5e57fff700 1 --2- 172.21.15.148:0/4145650720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e580f62a0 0x7f5e580f66c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.991+0000 7f5e57fff700 1 -- 172.21.15.148:0/4145650720 >> 172.21.15.148:0/4145650720 conn(0x7f5e580f1ac0 msgr2=0x7f5e580f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.992+0000 7f5e57fff700 1 -- 172.21.15.148:0/4145650720 shutdown_connections 2023-12-21T03:42:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.992+0000 7f5e57fff700 1 -- 172.21.15.148:0/4145650720 wait complete. 2023-12-21T03:42:25.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.992+0000 7f5e57fff700 1 Processor -- start 2023-12-21T03:42:25.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.992+0000 7f5e57fff700 1 -- start start 2023-12-21T03:42:25.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.992+0000 7f5e57fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e580f62a0 0x7f5e5806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:25.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.992+0000 7f5e57fff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e58069e20 con 0x7f5e580f62a0 2023-12-21T03:42:25.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.993+0000 7f5e56ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e580f62a0 0x7f5e5806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:25.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.993+0000 7f5e56ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e580f62a0 0x7f5e5806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:50912/0 (socket says 172.21.15.148:50912) 2023-12-21T03:42:25.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.993+0000 7f5e56ffd700 1 -- 172.21.15.148:0/1094638250 learned_addr learned my addr 172.21.15.148:0/1094638250 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:25.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.993+0000 7f5e56ffd700 1 -- 172.21.15.148:0/1094638250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e48005040 con 0x7f5e580f62a0 2023-12-21T03:42:25.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.993+0000 7f5e56ffd700 1 --2- 172.21.15.148:0/1094638250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e580f62a0 0x7f5e5806b9a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5e48013510 tx=0x7f5e4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:25.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.993+0000 7f5e3f7fe700 1 -- 172.21.15.148:0/1094638250 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e48009320 con 0x7f5e580f62a0 2023-12-21T03:42:25.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.993+0000 7f5e57fff700 1 -- 172.21.15.148:0/1094638250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e58069fc0 con 0x7f5e580f62a0 2023-12-21T03:42:25.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.993+0000 7f5e3f7fe700 1 -- 172.21.15.148:0/1094638250 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e48013f00 con 0x7f5e580f62a0 2023-12-21T03:42:25.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.993+0000 7f5e3f7fe700 1 -- 172.21.15.148:0/1094638250 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e48014dc0 con 0x7f5e580f62a0 2023-12-21T03:42:25.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.993+0000 7f5e57fff700 1 -- 172.21.15.148:0/1094638250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5806a450 con 0x7f5e580f62a0 2023-12-21T03:42:25.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.994+0000 7f5e3f7fe700 1 -- 172.21.15.148:0/1094638250 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e480095f0 con 0x7f5e580f62a0 2023-12-21T03:42:25.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.994+0000 7f5e57fff700 1 -- 172.21.15.148:0/1094638250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e58059100 con 0x7f5e580f62a0 2023-12-21T03:42:25.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.994+0000 7f5e3f7fe700 1 --2- 172.21.15.148:0/1094638250 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e4002ff60 0x7f5e40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:25.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.994+0000 7f5e3f7fe700 1 -- 172.21.15.148:0/1094638250 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e48041040 con 0x7f5e580f62a0 2023-12-21T03:42:25.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.995+0000 7f5e567fc700 1 --2- 172.21.15.148:0/1094638250 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e4002ff60 0x7f5e40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:25.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:25.995+0000 7f5e567fc700 1 --2- 172.21.15.148:0/1094638250 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e4002ff60 0x7f5e40032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f5e4c0096a0 tx=0x7f5e4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:26.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.000+0000 7f5e3f7fe700 1 -- 172.21.15.148:0/1094638250 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e48014420 con 0x7f5e580f62a0 2023-12-21T03:42:26.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.204+0000 7f5e57fff700 1 -- 172.21.15.148:0/1094638250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e580fbce0 con 0x7f5e580f62a0 2023-12-21T03:42:26.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.205+0000 7f5e3f7fe700 1 -- 172.21.15.148:0/1094638250 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e48010230 con 0x7f5e580f62a0 2023-12-21T03:42:26.207 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:26.207 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:26.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.206+0000 7f5e57fff700 1 -- 172.21.15.148:0/1094638250 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e4002ff60 msgr2=0x7f5e40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:26.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.206+0000 7f5e57fff700 1 --2- 172.21.15.148:0/1094638250 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e4002ff60 0x7f5e40032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f5e4c0096a0 tx=0x7f5e4c006b40).stop 2023-12-21T03:42:26.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.206+0000 7f5e57fff700 1 -- 172.21.15.148:0/1094638250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e580f62a0 msgr2=0x7f5e5806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:26.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.206+0000 7f5e57fff700 1 --2- 172.21.15.148:0/1094638250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e580f62a0 0x7f5e5806b9a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5e48013510 tx=0x7f5e4800b040).stop 2023-12-21T03:42:26.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.206+0000 7f5e57fff700 1 -- 172.21.15.148:0/1094638250 shutdown_connections 2023-12-21T03:42:26.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.206+0000 7f5e57fff700 1 --2- 172.21.15.148:0/1094638250 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e4002ff60 0x7f5e40032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:26.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.206+0000 7f5e57fff700 1 --2- 172.21.15.148:0/1094638250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e580f62a0 0x7f5e5806b9a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:26.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.206+0000 7f5e57fff700 1 -- 172.21.15.148:0/1094638250 >> 172.21.15.148:0/1094638250 conn(0x7f5e580f1ac0 msgr2=0x7f5e580f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:26.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.206+0000 7f5e57fff700 1 -- 172.21.15.148:0/1094638250 shutdown_connections 2023-12-21T03:42:26.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:26.206+0000 7f5e57fff700 1 -- 172.21.15.148:0/1094638250 wait complete. 2023-12-21T03:42:26.209 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:26.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:26.206470+0000 mon.smithi078 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.148:0/1094638250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:27.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:26.429018+0000 mgr.smithi078.zidbgq (mgr.14162) 95 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:27.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:27.595 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.776+0000 7f869031a700 1 -- 172.21.15.148:0/2837817392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86880f6510 msgr2=0x7f86880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.776+0000 7f869031a700 1 --2- 172.21.15.148:0/2837817392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86880f6510 0x7f86880f6930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f8678009fb0 tx=0x7f8678009b30).stop 2023-12-21T03:42:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.776+0000 7f869031a700 1 -- 172.21.15.148:0/2837817392 shutdown_connections 2023-12-21T03:42:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.776+0000 7f869031a700 1 --2- 172.21.15.148:0/2837817392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86880f6510 0x7f86880f6930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.776+0000 7f869031a700 1 -- 172.21.15.148:0/2837817392 >> 172.21.15.148:0/2837817392 conn(0x7f86880f1c70 msgr2=0x7f86880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.776+0000 7f869031a700 1 -- 172.21.15.148:0/2837817392 shutdown_connections 2023-12-21T03:42:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.776+0000 7f869031a700 1 -- 172.21.15.148:0/2837817392 wait complete. 2023-12-21T03:42:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.777+0000 7f869031a700 1 Processor -- start 2023-12-21T03:42:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.777+0000 7f869031a700 1 -- start start 2023-12-21T03:42:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.777+0000 7f869031a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86880f6510 0x7f8688106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.777+0000 7f869031a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688106c20 con 0x7f86880f6510 2023-12-21T03:42:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.777+0000 7f868e0b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86880f6510 0x7f8688106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.777+0000 7f868e0b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86880f6510 0x7f8688106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:50926/0 (socket says 172.21.15.148:50926) 2023-12-21T03:42:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.777+0000 7f868e0b6700 1 -- 172.21.15.148:0/1028105816 learned_addr learned my addr 172.21.15.148:0/1028105816 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.777+0000 7f868e0b6700 1 -- 172.21.15.148:0/1028105816 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8678005040 con 0x7f86880f6510 2023-12-21T03:42:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.777+0000 7f868e0b6700 1 --2- 172.21.15.148:0/1028105816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86880f6510 0x7f8688106730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f8678013510 tx=0x7f867800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.778+0000 7f867effd700 1 -- 172.21.15.148:0/1028105816 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8678009320 con 0x7f86880f6510 2023-12-21T03:42:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.778+0000 7f869031a700 1 -- 172.21.15.148:0/1028105816 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8688106e20 con 0x7f86880f6510 2023-12-21T03:42:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.778+0000 7f867effd700 1 -- 172.21.15.148:0/1028105816 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8678013f00 con 0x7f86880f6510 2023-12-21T03:42:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.778+0000 7f867effd700 1 -- 172.21.15.148:0/1028105816 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8678014d90 con 0x7f86880f6510 2023-12-21T03:42:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.778+0000 7f869031a700 1 -- 172.21.15.148:0/1028105816 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86881072b0 con 0x7f86880f6510 2023-12-21T03:42:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.778+0000 7f867effd700 1 -- 172.21.15.148:0/1028105816 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86780095f0 con 0x7f86880f6510 2023-12-21T03:42:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.778+0000 7f869031a700 1 -- 172.21.15.148:0/1028105816 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8688045ad0 con 0x7f86880f6510 2023-12-21T03:42:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.779+0000 7f867effd700 1 --2- 172.21.15.148:0/1028105816 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f867402ff60 0x7f8674032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.779+0000 7f867effd700 1 -- 172.21.15.148:0/1028105816 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86780410c0 con 0x7f86880f6510 2023-12-21T03:42:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.779+0000 7f868d8b5700 1 --2- 172.21.15.148:0/1028105816 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f867402ff60 0x7f8674032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.779+0000 7f868d8b5700 1 --2- 172.21.15.148:0/1028105816 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f867402ff60 0x7f8674032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f86840096a0 tx=0x7f8684006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.784+0000 7f867effd700 1 -- 172.21.15.148:0/1028105816 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8678014420 con 0x7f86880f6510 2023-12-21T03:42:28.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:28.429270+0000 mgr.smithi078.zidbgq (mgr.14162) 96 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:28.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.987+0000 7f869031a700 1 -- 172.21.15.148:0/1028105816 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86880fbf60 con 0x7f86880f6510 2023-12-21T03:42:28.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.987+0000 7f867effd700 1 -- 172.21.15.148:0/1028105816 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8678010230 con 0x7f86880f6510 2023-12-21T03:42:28.990 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:28.990 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:28.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.989+0000 7f869031a700 1 -- 172.21.15.148:0/1028105816 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f867402ff60 msgr2=0x7f8674032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:28.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.989+0000 7f869031a700 1 --2- 172.21.15.148:0/1028105816 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f867402ff60 0x7f8674032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f86840096a0 tx=0x7f8684006b40).stop 2023-12-21T03:42:28.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.989+0000 7f869031a700 1 -- 172.21.15.148:0/1028105816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86880f6510 msgr2=0x7f8688106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:28.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.989+0000 7f869031a700 1 --2- 172.21.15.148:0/1028105816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86880f6510 0x7f8688106730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f8678013510 tx=0x7f867800b040).stop 2023-12-21T03:42:28.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.989+0000 7f869031a700 1 -- 172.21.15.148:0/1028105816 shutdown_connections 2023-12-21T03:42:28.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.989+0000 7f869031a700 1 --2- 172.21.15.148:0/1028105816 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f867402ff60 0x7f8674032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:28.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.989+0000 7f869031a700 1 --2- 172.21.15.148:0/1028105816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86880f6510 0x7f8688106730 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:28.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.989+0000 7f869031a700 1 -- 172.21.15.148:0/1028105816 >> 172.21.15.148:0/1028105816 conn(0x7f86880f1c70 msgr2=0x7f86880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:28.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.989+0000 7f869031a700 1 -- 172.21.15.148:0/1028105816 shutdown_connections 2023-12-21T03:42:28.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:28.989+0000 7f869031a700 1 -- 172.21.15.148:0/1028105816 wait complete. 2023-12-21T03:42:28.992 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:28.989176+0000 mon.smithi078 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.148:0/1028105816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:30.384 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:30.384 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:30.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:30.429534+0000 mgr.smithi078.zidbgq (mgr.14162) 97 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:31.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.493+0000 7fe4681ba700 1 -- 172.21.15.148:0/3768790902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4600f5410 msgr2=0x7fe4600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:31.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.493+0000 7fe4681ba700 1 --2- 172.21.15.148:0/3768790902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4600f5410 0x7fe4600f5830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe45c009fb0 tx=0x7fe45c009b30).stop 2023-12-21T03:42:31.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.493+0000 7fe4681ba700 1 -- 172.21.15.148:0/3768790902 shutdown_connections 2023-12-21T03:42:31.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.493+0000 7fe4681ba700 1 --2- 172.21.15.148:0/3768790902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4600f5410 0x7fe4600f5830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:31.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.493+0000 7fe4681ba700 1 -- 172.21.15.148:0/3768790902 >> 172.21.15.148:0/3768790902 conn(0x7fe4600f0b70 msgr2=0x7fe4600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:31.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.493+0000 7fe4681ba700 1 -- 172.21.15.148:0/3768790902 shutdown_connections 2023-12-21T03:42:31.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.493+0000 7fe4681ba700 1 -- 172.21.15.148:0/3768790902 wait complete. 2023-12-21T03:42:31.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.494+0000 7fe4681ba700 1 Processor -- start 2023-12-21T03:42:31.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.494+0000 7fe4681ba700 1 -- start start 2023-12-21T03:42:31.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.494+0000 7fe4681ba700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4600f5410 0x7fe4601093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:31.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.494+0000 7fe4681ba700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe460060280 con 0x7fe4600f5410 2023-12-21T03:42:31.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.494+0000 7fe465f56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4600f5410 0x7fe4601093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:31.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.494+0000 7fe465f56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4600f5410 0x7fe4601093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:55776/0 (socket says 172.21.15.148:55776) 2023-12-21T03:42:31.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.494+0000 7fe465f56700 1 -- 172.21.15.148:0/3225912659 learned_addr learned my addr 172.21.15.148:0/3225912659 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:31.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.494+0000 7fe465f56700 1 -- 172.21.15.148:0/3225912659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c005040 con 0x7fe4600f5410 2023-12-21T03:42:31.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.494+0000 7fe465f56700 1 --2- 172.21.15.148:0/3225912659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4600f5410 0x7fe4601093f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fe45c013510 tx=0x7fe45c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:31.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.494+0000 7fe4567fc700 1 -- 172.21.15.148:0/3225912659 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe45c009320 con 0x7fe4600f5410 2023-12-21T03:42:31.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.495+0000 7fe4681ba700 1 -- 172.21.15.148:0/3225912659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe460060480 con 0x7fe4600f5410 2023-12-21T03:42:31.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.495+0000 7fe4567fc700 1 -- 172.21.15.148:0/3225912659 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe45c013f00 con 0x7fe4600f5410 2023-12-21T03:42:31.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.495+0000 7fe4567fc700 1 -- 172.21.15.148:0/3225912659 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe45c014d90 con 0x7fe4600f5410 2023-12-21T03:42:31.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.495+0000 7fe4681ba700 1 -- 172.21.15.148:0/3225912659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe460109b20 con 0x7fe4600f5410 2023-12-21T03:42:31.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.495+0000 7fe4681ba700 1 -- 172.21.15.148:0/3225912659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe460045ad0 con 0x7fe4600f5410 2023-12-21T03:42:31.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.495+0000 7fe4567fc700 1 -- 172.21.15.148:0/3225912659 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe45c0095f0 con 0x7fe4600f5410 2023-12-21T03:42:31.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.495+0000 7fe4567fc700 1 --2- 172.21.15.148:0/3225912659 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe44c02ff60 0x7fe44c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:31.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.496+0000 7fe4567fc700 1 -- 172.21.15.148:0/3225912659 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe45c040fe0 con 0x7fe4600f5410 2023-12-21T03:42:31.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.496+0000 7fe465755700 1 --2- 172.21.15.148:0/3225912659 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe44c02ff60 0x7fe44c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:31.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.496+0000 7fe465755700 1 --2- 172.21.15.148:0/3225912659 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe44c02ff60 0x7fe44c032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe4500096a0 tx=0x7fe450006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:31.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.501+0000 7fe4567fc700 1 -- 172.21.15.148:0/3225912659 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe45c01d7f0 con 0x7fe4600f5410 2023-12-21T03:42:31.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.706+0000 7fe4681ba700 1 -- 172.21.15.148:0/3225912659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe46010a340 con 0x7fe4600f5410 2023-12-21T03:42:31.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.706+0000 7fe4567fc700 1 -- 172.21.15.148:0/3225912659 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe45c010230 con 0x7fe4600f5410 2023-12-21T03:42:31.708 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:31.709 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:31.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.708+0000 7fe4681ba700 1 -- 172.21.15.148:0/3225912659 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe44c02ff60 msgr2=0x7fe44c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:31.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.708+0000 7fe4681ba700 1 --2- 172.21.15.148:0/3225912659 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe44c02ff60 0x7fe44c032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe4500096a0 tx=0x7fe450006b40).stop 2023-12-21T03:42:31.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.708+0000 7fe4681ba700 1 -- 172.21.15.148:0/3225912659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4600f5410 msgr2=0x7fe4601093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:31.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.708+0000 7fe4681ba700 1 --2- 172.21.15.148:0/3225912659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4600f5410 0x7fe4601093f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fe45c013510 tx=0x7fe45c00b040).stop 2023-12-21T03:42:31.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.708+0000 7fe4681ba700 1 -- 172.21.15.148:0/3225912659 shutdown_connections 2023-12-21T03:42:31.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.708+0000 7fe4681ba700 1 --2- 172.21.15.148:0/3225912659 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe44c02ff60 0x7fe44c032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:31.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.708+0000 7fe4681ba700 1 --2- 172.21.15.148:0/3225912659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4600f5410 0x7fe4601093f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:31.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.708+0000 7fe4681ba700 1 -- 172.21.15.148:0/3225912659 >> 172.21.15.148:0/3225912659 conn(0x7fe4600f0b70 msgr2=0x7fe4600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:31.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.708+0000 7fe4681ba700 1 -- 172.21.15.148:0/3225912659 shutdown_connections 2023-12-21T03:42:31.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:31.708+0000 7fe4681ba700 1 -- 172.21.15.148:0/3225912659 wait complete. 2023-12-21T03:42:31.711 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:30.542661+0000 mon.smithi078 (mon.0) 184 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:42:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:30.550353+0000 mon.smithi078 (mon.0) 185 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:42:32.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:31.708127+0000 mon.smithi078 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.148:0/3225912659' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:32.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:32.429893+0000 mgr.smithi078.zidbgq (mgr.14162) 98 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:33.088 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:33.088 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:34.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.135+0000 7f711a04c700 1 -- 172.21.15.148:0/1555994410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71140f5410 msgr2=0x7f71140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:34.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.135+0000 7f711a04c700 1 --2- 172.21.15.148:0/1555994410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71140f5410 0x7f71140f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f7108009fb0 tx=0x7f7108009b30).stop 2023-12-21T03:42:34.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.135+0000 7f711a04c700 1 -- 172.21.15.148:0/1555994410 shutdown_connections 2023-12-21T03:42:34.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.135+0000 7f711a04c700 1 --2- 172.21.15.148:0/1555994410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71140f5410 0x7f71140f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:34.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.135+0000 7f711a04c700 1 -- 172.21.15.148:0/1555994410 >> 172.21.15.148:0/1555994410 conn(0x7f71140f0b70 msgr2=0x7f71140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:34.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.135+0000 7f711a04c700 1 -- 172.21.15.148:0/1555994410 shutdown_connections 2023-12-21T03:42:34.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.135+0000 7f711a04c700 1 -- 172.21.15.148:0/1555994410 wait complete. 2023-12-21T03:42:34.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.136+0000 7f711a04c700 1 Processor -- start 2023-12-21T03:42:34.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.136+0000 7f711a04c700 1 -- start start 2023-12-21T03:42:34.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.136+0000 7f711a04c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71140f5410 0x7f7114109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:34.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.136+0000 7f711a04c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7114060240 con 0x7f71140f5410 2023-12-21T03:42:34.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.136+0000 7f71137fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71140f5410 0x7f7114109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:34.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.136+0000 7f71137fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71140f5410 0x7f7114109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:55806/0 (socket says 172.21.15.148:55806) 2023-12-21T03:42:34.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.136+0000 7f71137fe700 1 -- 172.21.15.148:0/1982655953 learned_addr learned my addr 172.21.15.148:0/1982655953 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:34.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.137+0000 7f71137fe700 1 -- 172.21.15.148:0/1982655953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7108005040 con 0x7f71140f5410 2023-12-21T03:42:34.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.137+0000 7f71137fe700 1 --2- 172.21.15.148:0/1982655953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71140f5410 0x7f7114109430 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7108013510 tx=0x7f710800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:34.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.137+0000 7f70fbfff700 1 -- 172.21.15.148:0/1982655953 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7108009320 con 0x7f71140f5410 2023-12-21T03:42:34.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.137+0000 7f711a04c700 1 -- 172.21.15.148:0/1982655953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7114060440 con 0x7f71140f5410 2023-12-21T03:42:34.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.137+0000 7f70fbfff700 1 -- 172.21.15.148:0/1982655953 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7108013f00 con 0x7f71140f5410 2023-12-21T03:42:34.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.137+0000 7f70fbfff700 1 -- 172.21.15.148:0/1982655953 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7108014d90 con 0x7f71140f5410 2023-12-21T03:42:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.137+0000 7f711a04c700 1 -- 172.21.15.148:0/1982655953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7114109ae0 con 0x7f71140f5410 2023-12-21T03:42:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.138+0000 7f70fbfff700 1 -- 172.21.15.148:0/1982655953 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71080095f0 con 0x7f71140f5410 2023-12-21T03:42:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.138+0000 7f711a04c700 1 -- 172.21.15.148:0/1982655953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7114045ad0 con 0x7f71140f5410 2023-12-21T03:42:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.138+0000 7f70fbfff700 1 --2- 172.21.15.148:0/1982655953 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70fc02ff60 0x7f70fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.138+0000 7f70fbfff700 1 -- 172.21.15.148:0/1982655953 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71080410c0 con 0x7f71140f5410 2023-12-21T03:42:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.138+0000 7f7112ffd700 1 --2- 172.21.15.148:0/1982655953 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70fc02ff60 0x7f70fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.138+0000 7f7112ffd700 1 --2- 172.21.15.148:0/1982655953 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70fc02ff60 0x7f70fc032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f71040096a0 tx=0x7f7104006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:34.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.142+0000 7f70fbfff700 1 -- 172.21.15.148:0/1982655953 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7108014420 con 0x7f71140f5410 2023-12-21T03:42:34.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.348+0000 7f711a04c700 1 -- 172.21.15.148:0/1982655953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71140fae60 con 0x7f71140f5410 2023-12-21T03:42:34.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.348+0000 7f70fbfff700 1 -- 172.21.15.148:0/1982655953 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7108010230 con 0x7f71140f5410 2023-12-21T03:42:34.350 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:34.350 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:34.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.350+0000 7f711a04c700 1 -- 172.21.15.148:0/1982655953 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70fc02ff60 msgr2=0x7f70fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:34.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.350+0000 7f711a04c700 1 --2- 172.21.15.148:0/1982655953 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70fc02ff60 0x7f70fc032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f71040096a0 tx=0x7f7104006b40).stop 2023-12-21T03:42:34.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.350+0000 7f711a04c700 1 -- 172.21.15.148:0/1982655953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71140f5410 msgr2=0x7f7114109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:34.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.350+0000 7f711a04c700 1 --2- 172.21.15.148:0/1982655953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71140f5410 0x7f7114109430 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7108013510 tx=0x7f710800b040).stop 2023-12-21T03:42:34.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.350+0000 7f711a04c700 1 -- 172.21.15.148:0/1982655953 shutdown_connections 2023-12-21T03:42:34.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.350+0000 7f711a04c700 1 --2- 172.21.15.148:0/1982655953 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70fc02ff60 0x7f70fc032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:34.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.350+0000 7f711a04c700 1 --2- 172.21.15.148:0/1982655953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71140f5410 0x7f7114109430 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:34.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.350+0000 7f711a04c700 1 -- 172.21.15.148:0/1982655953 >> 172.21.15.148:0/1982655953 conn(0x7f71140f0b70 msgr2=0x7f71140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:34.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.350+0000 7f711a04c700 1 -- 172.21.15.148:0/1982655953 shutdown_connections 2023-12-21T03:42:34.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:34.350+0000 7f711a04c700 1 -- 172.21.15.148:0/1982655953 wait complete. 2023-12-21T03:42:34.352 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:35.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:34.349386+0000 mon.smithi078 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.148:0/1982655953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:35.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:34.430284+0000 mgr.smithi078.zidbgq (mgr.14162) 99 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:35.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:35.761 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:36.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:36 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:36.430558+0000 mgr.smithi078.zidbgq (mgr.14162) 100 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:36.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.921+0000 7f527675d700 1 -- 172.21.15.148:0/2312580317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52700f62a0 msgr2=0x7f52700f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:36.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.921+0000 7f527675d700 1 --2- 172.21.15.148:0/2312580317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52700f62a0 0x7f52700f66c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f5260009fb0 tx=0x7f5260009b30).stop 2023-12-21T03:42:36.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.921+0000 7f527675d700 1 -- 172.21.15.148:0/2312580317 shutdown_connections 2023-12-21T03:42:36.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.921+0000 7f527675d700 1 --2- 172.21.15.148:0/2312580317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52700f62a0 0x7f52700f66c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:36.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.922+0000 7f527675d700 1 -- 172.21.15.148:0/2312580317 >> 172.21.15.148:0/2312580317 conn(0x7f52700f1ac0 msgr2=0x7f52700f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:36.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.922+0000 7f527675d700 1 -- 172.21.15.148:0/2312580317 shutdown_connections 2023-12-21T03:42:36.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.922+0000 7f527675d700 1 -- 172.21.15.148:0/2312580317 wait complete. 2023-12-21T03:42:36.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.922+0000 7f527675d700 1 Processor -- start 2023-12-21T03:42:36.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.922+0000 7f527675d700 1 -- start start 2023-12-21T03:42:36.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.922+0000 7f527675d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52700f62a0 0x7f5270106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:36.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.923+0000 7f527675d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52701069f0 con 0x7f52700f62a0 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.923+0000 7f527575b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52700f62a0 0x7f5270106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.923+0000 7f527575b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52700f62a0 0x7f5270106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:55818/0 (socket says 172.21.15.148:55818) 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.923+0000 7f527575b700 1 -- 172.21.15.148:0/2040544485 learned_addr learned my addr 172.21.15.148:0/2040544485 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.923+0000 7f527575b700 1 -- 172.21.15.148:0/2040544485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5260005040 con 0x7f52700f62a0 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.923+0000 7f527575b700 1 --2- 172.21.15.148:0/2040544485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52700f62a0 0x7f5270106500 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f5260013510 tx=0x7f526000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.923+0000 7f5265ffb700 1 -- 172.21.15.148:0/2040544485 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5260009320 con 0x7f52700f62a0 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.923+0000 7f527675d700 1 -- 172.21.15.148:0/2040544485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5270106bf0 con 0x7f52700f62a0 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.923+0000 7f5265ffb700 1 -- 172.21.15.148:0/2040544485 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5260013f00 con 0x7f52700f62a0 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.923+0000 7f5265ffb700 1 -- 172.21.15.148:0/2040544485 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5260014dc0 con 0x7f52700f62a0 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.924+0000 7f527675d700 1 -- 172.21.15.148:0/2040544485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5270107080 con 0x7f52700f62a0 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.924+0000 7f5265ffb700 1 -- 172.21.15.148:0/2040544485 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52600095f0 con 0x7f52700f62a0 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.924+0000 7f527675d700 1 -- 172.21.15.148:0/2040544485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5270045ad0 con 0x7f52700f62a0 2023-12-21T03:42:36.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.924+0000 7f5265ffb700 1 --2- 172.21.15.148:0/2040544485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f525c02ff60 0x7f525c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:36.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.925+0000 7f5265ffb700 1 -- 172.21.15.148:0/2040544485 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5260041100 con 0x7f52700f62a0 2023-12-21T03:42:36.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.925+0000 7f5274f5a700 1 --2- 172.21.15.148:0/2040544485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f525c02ff60 0x7f525c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:36.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.925+0000 7f5274f5a700 1 --2- 172.21.15.148:0/2040544485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f525c02ff60 0x7f525c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f526c0096a0 tx=0x7f526c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:36.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:36.929+0000 7f5265ffb700 1 -- 172.21.15.148:0/2040544485 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5260014420 con 0x7f52700f62a0 2023-12-21T03:42:37.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.134+0000 7f527675d700 1 -- 172.21.15.148:0/2040544485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5270107f30 con 0x7f52700f62a0 2023-12-21T03:42:37.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.135+0000 7f5265ffb700 1 -- 172.21.15.148:0/2040544485 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5260010230 con 0x7f52700f62a0 2023-12-21T03:42:37.136 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:37.136 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.137+0000 7f527675d700 1 -- 172.21.15.148:0/2040544485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f525c02ff60 msgr2=0x7f525c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.137+0000 7f527675d700 1 --2- 172.21.15.148:0/2040544485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f525c02ff60 0x7f525c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f526c0096a0 tx=0x7f526c006b40).stop 2023-12-21T03:42:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.137+0000 7f527675d700 1 -- 172.21.15.148:0/2040544485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52700f62a0 msgr2=0x7f5270106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.137+0000 7f527675d700 1 --2- 172.21.15.148:0/2040544485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52700f62a0 0x7f5270106500 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f5260013510 tx=0x7f526000b040).stop 2023-12-21T03:42:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.137+0000 7f527675d700 1 -- 172.21.15.148:0/2040544485 shutdown_connections 2023-12-21T03:42:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.137+0000 7f527675d700 1 --2- 172.21.15.148:0/2040544485 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f525c02ff60 0x7f525c032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.137+0000 7f527675d700 1 --2- 172.21.15.148:0/2040544485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52700f62a0 0x7f5270106500 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.137+0000 7f527675d700 1 -- 172.21.15.148:0/2040544485 >> 172.21.15.148:0/2040544485 conn(0x7f52700f1ac0 msgr2=0x7f52700f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.137+0000 7f527675d700 1 -- 172.21.15.148:0/2040544485 shutdown_connections 2023-12-21T03:42:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:37.137+0000 7f527675d700 1 -- 172.21.15.148:0/2040544485 wait complete. 2023-12-21T03:42:37.139 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:37.135971+0000 mon.smithi078 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.148:0/2040544485' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:38.512 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:38.513 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:38.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:38.430943+0000 mgr.smithi078.zidbgq (mgr.14162) 101 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:39.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.712+0000 7fb97ef30700 1 -- 172.21.15.148:0/192897074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9780f51a0 msgr2=0x7fb9780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:39.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.712+0000 7fb97ef30700 1 --2- 172.21.15.148:0/192897074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9780f51a0 0x7fb9780f55c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fb968009fb0 tx=0x7fb968009b30).stop 2023-12-21T03:42:39.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.712+0000 7fb97ef30700 1 -- 172.21.15.148:0/192897074 shutdown_connections 2023-12-21T03:42:39.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.712+0000 7fb97ef30700 1 --2- 172.21.15.148:0/192897074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9780f51a0 0x7fb9780f55c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:39.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.712+0000 7fb97ef30700 1 -- 172.21.15.148:0/192897074 >> 172.21.15.148:0/192897074 conn(0x7fb9780f09a0 msgr2=0x7fb9780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:39.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.712+0000 7fb97ef30700 1 -- 172.21.15.148:0/192897074 shutdown_connections 2023-12-21T03:42:39.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.713+0000 7fb97ef30700 1 -- 172.21.15.148:0/192897074 wait complete. 2023-12-21T03:42:39.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.713+0000 7fb97ef30700 1 Processor -- start 2023-12-21T03:42:39.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.713+0000 7fb97ef30700 1 -- start start 2023-12-21T03:42:39.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.713+0000 7fb97ef30700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9780f51a0 0x7fb9781053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:39.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.713+0000 7fb97ef30700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9781058a0 con 0x7fb9780f51a0 2023-12-21T03:42:39.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.714+0000 7fb97df2e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9780f51a0 0x7fb9781053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:39.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.714+0000 7fb97df2e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9780f51a0 0x7fb9781053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:45352/0 (socket says 172.21.15.148:45352) 2023-12-21T03:42:39.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.714+0000 7fb97df2e700 1 -- 172.21.15.148:0/4235315701 learned_addr learned my addr 172.21.15.148:0/4235315701 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:39.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.714+0000 7fb97df2e700 1 -- 172.21.15.148:0/4235315701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb968005040 con 0x7fb9780f51a0 2023-12-21T03:42:39.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.714+0000 7fb97df2e700 1 --2- 172.21.15.148:0/4235315701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9780f51a0 0x7fb9781053b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fb968013510 tx=0x7fb96800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:39.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.714+0000 7fb96e7fc700 1 -- 172.21.15.148:0/4235315701 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb968009320 con 0x7fb9780f51a0 2023-12-21T03:42:39.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.714+0000 7fb97ef30700 1 -- 172.21.15.148:0/4235315701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb978105aa0 con 0x7fb9780f51a0 2023-12-21T03:42:39.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.714+0000 7fb96e7fc700 1 -- 172.21.15.148:0/4235315701 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb968013f00 con 0x7fb9780f51a0 2023-12-21T03:42:39.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.714+0000 7fb96e7fc700 1 -- 172.21.15.148:0/4235315701 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb968014dc0 con 0x7fb9780f51a0 2023-12-21T03:42:39.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.714+0000 7fb97ef30700 1 -- 172.21.15.148:0/4235315701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb978105f30 con 0x7fb9780f51a0 2023-12-21T03:42:39.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.715+0000 7fb96e7fc700 1 -- 172.21.15.148:0/4235315701 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9680263a0 con 0x7fb9780f51a0 2023-12-21T03:42:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.715+0000 7fb97ef30700 1 -- 172.21.15.148:0/4235315701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb978045ad0 con 0x7fb9780f51a0 2023-12-21T03:42:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.716+0000 7fb96e7fc700 1 --2- 172.21.15.148:0/4235315701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb96402ffb0 0x7fb964032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.716+0000 7fb96e7fc700 1 -- 172.21.15.148:0/4235315701 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9680412e0 con 0x7fb9780f51a0 2023-12-21T03:42:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.716+0000 7fb97d72d700 1 --2- 172.21.15.148:0/4235315701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb96402ffb0 0x7fb964032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.716+0000 7fb97d72d700 1 --2- 172.21.15.148:0/4235315701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb96402ffb0 0x7fb964032470 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fb9740096a0 tx=0x7fb974006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.720+0000 7fb96e7fc700 1 -- 172.21.15.148:0/4235315701 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb968014420 con 0x7fb9780f51a0 2023-12-21T03:42:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.923+0000 7fb97ef30700 1 -- 172.21.15.148:0/4235315701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb978106d80 con 0x7fb9780f51a0 2023-12-21T03:42:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.924+0000 7fb96e7fc700 1 -- 172.21.15.148:0/4235315701 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb96800f340 con 0x7fb9780f51a0 2023-12-21T03:42:39.925 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:39.925 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.925+0000 7fb97ef30700 1 -- 172.21.15.148:0/4235315701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb96402ffb0 msgr2=0x7fb964032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.925+0000 7fb97ef30700 1 --2- 172.21.15.148:0/4235315701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb96402ffb0 0x7fb964032470 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fb9740096a0 tx=0x7fb974006b40).stop 2023-12-21T03:42:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.925+0000 7fb97ef30700 1 -- 172.21.15.148:0/4235315701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9780f51a0 msgr2=0x7fb9781053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.925+0000 7fb97ef30700 1 --2- 172.21.15.148:0/4235315701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9780f51a0 0x7fb9781053b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fb968013510 tx=0x7fb96800b040).stop 2023-12-21T03:42:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.925+0000 7fb97ef30700 1 -- 172.21.15.148:0/4235315701 shutdown_connections 2023-12-21T03:42:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.925+0000 7fb97ef30700 1 --2- 172.21.15.148:0/4235315701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb96402ffb0 0x7fb964032470 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.925+0000 7fb97ef30700 1 --2- 172.21.15.148:0/4235315701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9780f51a0 0x7fb9781053b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.925+0000 7fb97ef30700 1 -- 172.21.15.148:0/4235315701 >> 172.21.15.148:0/4235315701 conn(0x7fb9780f09a0 msgr2=0x7fb9780f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.926+0000 7fb97ef30700 1 -- 172.21.15.148:0/4235315701 shutdown_connections 2023-12-21T03:42:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:39.926+0000 7fb97ef30700 1 -- 172.21.15.148:0/4235315701 wait complete. 2023-12-21T03:42:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:40.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:39.924594+0000 mon.smithi078 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.148:0/4235315701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:41.296 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:41.296 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:41.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:40.431228+0000 mgr.smithi078.zidbgq (mgr.14162) 102 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:42.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.417+0000 7f1518928700 1 -- 172.21.15.148:0/1507509106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15140f6510 msgr2=0x7f15140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:42.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.417+0000 7f1518928700 1 --2- 172.21.15.148:0/1507509106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15140f6510 0x7f15140f6930 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f1504009fb0 tx=0x7f1504009b30).stop 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.417+0000 7f1518928700 1 -- 172.21.15.148:0/1507509106 shutdown_connections 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.417+0000 7f1518928700 1 --2- 172.21.15.148:0/1507509106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15140f6510 0x7f15140f6930 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.417+0000 7f1518928700 1 -- 172.21.15.148:0/1507509106 >> 172.21.15.148:0/1507509106 conn(0x7f15140f1c70 msgr2=0x7f15140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.417+0000 7f1518928700 1 -- 172.21.15.148:0/1507509106 shutdown_connections 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.417+0000 7f1518928700 1 -- 172.21.15.148:0/1507509106 wait complete. 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.418+0000 7f1518928700 1 Processor -- start 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.418+0000 7f1518928700 1 -- start start 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.418+0000 7f1518928700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15140f6510 0x7f1514106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.418+0000 7f1518928700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1514106c20 con 0x7f15140f6510 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.418+0000 7f151259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15140f6510 0x7f1514106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.418+0000 7f151259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15140f6510 0x7f1514106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:45358/0 (socket says 172.21.15.148:45358) 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.418+0000 7f151259c700 1 -- 172.21.15.148:0/3042977177 learned_addr learned my addr 172.21.15.148:0/3042977177 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.418+0000 7f151259c700 1 -- 172.21.15.148:0/3042977177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1504005040 con 0x7f15140f6510 2023-12-21T03:42:42.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.418+0000 7f151259c700 1 --2- 172.21.15.148:0/3042977177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15140f6510 0x7f1514106730 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1504013510 tx=0x7f150400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:42.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.419+0000 7f150affd700 1 -- 172.21.15.148:0/3042977177 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1504009320 con 0x7f15140f6510 2023-12-21T03:42:42.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.419+0000 7f150affd700 1 -- 172.21.15.148:0/3042977177 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1504013f00 con 0x7f15140f6510 2023-12-21T03:42:42.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.419+0000 7f1518928700 1 -- 172.21.15.148:0/3042977177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1514106e20 con 0x7f15140f6510 2023-12-21T03:42:42.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.419+0000 7f150affd700 1 -- 172.21.15.148:0/3042977177 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1504014d90 con 0x7f15140f6510 2023-12-21T03:42:42.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.419+0000 7f1518928700 1 -- 172.21.15.148:0/3042977177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15141072b0 con 0x7f15140f6510 2023-12-21T03:42:42.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.419+0000 7f150affd700 1 -- 172.21.15.148:0/3042977177 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1504014420 con 0x7f15140f6510 2023-12-21T03:42:42.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.419+0000 7f1518928700 1 -- 172.21.15.148:0/3042977177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1514045ad0 con 0x7f15140f6510 2023-12-21T03:42:42.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.420+0000 7f150affd700 1 --2- 172.21.15.148:0/3042977177 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f150002ff60 0x7f1500032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:42.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.420+0000 7f150affd700 1 -- 172.21.15.148:0/3042977177 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1504041050 con 0x7f15140f6510 2023-12-21T03:42:42.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.420+0000 7f1511d9b700 1 --2- 172.21.15.148:0/3042977177 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f150002ff60 0x7f1500032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:42.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.420+0000 7f1511d9b700 1 --2- 172.21.15.148:0/3042977177 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f150002ff60 0x7f1500032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f14fc0096a0 tx=0x7f14fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:42.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.424+0000 7f150affd700 1 -- 172.21.15.148:0/3042977177 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15040095f0 con 0x7f15140f6510 2023-12-21T03:42:42.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.633+0000 7f1518928700 1 -- 172.21.15.148:0/3042977177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15140fbf60 con 0x7f15140f6510 2023-12-21T03:42:42.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.634+0000 7f150affd700 1 -- 172.21.15.148:0/3042977177 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f150401db70 con 0x7f15140f6510 2023-12-21T03:42:42.635 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:42.635 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:42.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.635+0000 7f1518928700 1 -- 172.21.15.148:0/3042977177 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f150002ff60 msgr2=0x7f1500032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:42.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.635+0000 7f1518928700 1 --2- 172.21.15.148:0/3042977177 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f150002ff60 0x7f1500032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f14fc0096a0 tx=0x7f14fc006b40).stop 2023-12-21T03:42:42.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.635+0000 7f1518928700 1 -- 172.21.15.148:0/3042977177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15140f6510 msgr2=0x7f1514106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:42.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.635+0000 7f1518928700 1 --2- 172.21.15.148:0/3042977177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15140f6510 0x7f1514106730 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1504013510 tx=0x7f150400b040).stop 2023-12-21T03:42:42.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.635+0000 7f1518928700 1 -- 172.21.15.148:0/3042977177 shutdown_connections 2023-12-21T03:42:42.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.635+0000 7f1518928700 1 --2- 172.21.15.148:0/3042977177 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f150002ff60 0x7f1500032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:42.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.635+0000 7f1518928700 1 --2- 172.21.15.148:0/3042977177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15140f6510 0x7f1514106730 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:42.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.635+0000 7f1518928700 1 -- 172.21.15.148:0/3042977177 >> 172.21.15.148:0/3042977177 conn(0x7f15140f1c70 msgr2=0x7f15140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:42.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.635+0000 7f1518928700 1 -- 172.21.15.148:0/3042977177 shutdown_connections 2023-12-21T03:42:42.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:42.635+0000 7f1518928700 1 -- 172.21.15.148:0/3042977177 wait complete. 2023-12-21T03:42:42.637 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:42.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:42.431455+0000 mgr.smithi078.zidbgq (mgr.14162) 103 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:42.634361+0000 mon.smithi078 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.148:0/3042977177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:44.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:44.013 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:45.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.160+0000 7f1441161700 1 -- 172.21.15.148:0/1977929279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f143c0f6510 msgr2=0x7f143c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:45.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.160+0000 7f1441161700 1 --2- 172.21.15.148:0/1977929279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f143c0f6510 0x7f143c0f6930 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1424009fb0 tx=0x7f1424009b30).stop 2023-12-21T03:42:45.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.160+0000 7f1441161700 1 -- 172.21.15.148:0/1977929279 shutdown_connections 2023-12-21T03:42:45.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.160+0000 7f1441161700 1 --2- 172.21.15.148:0/1977929279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f143c0f6510 0x7f143c0f6930 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:45.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.160+0000 7f1441161700 1 -- 172.21.15.148:0/1977929279 >> 172.21.15.148:0/1977929279 conn(0x7f143c0f1c70 msgr2=0x7f143c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:45.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.160+0000 7f1441161700 1 -- 172.21.15.148:0/1977929279 shutdown_connections 2023-12-21T03:42:45.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.160+0000 7f1441161700 1 -- 172.21.15.148:0/1977929279 wait complete. 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.160+0000 7f1441161700 1 Processor -- start 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.161+0000 7f1441161700 1 -- start start 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.161+0000 7f1441161700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f143c0f6510 0x7f143c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.161+0000 7f1441161700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c106c20 con 0x7f143c0f6510 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.161+0000 7f143ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f143c0f6510 0x7f143c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.161+0000 7f143ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f143c0f6510 0x7f143c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:45368/0 (socket says 172.21.15.148:45368) 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.161+0000 7f143ad9d700 1 -- 172.21.15.148:0/1249258989 learned_addr learned my addr 172.21.15.148:0/1249258989 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.161+0000 7f143ad9d700 1 -- 172.21.15.148:0/1249258989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1424005040 con 0x7f143c0f6510 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.161+0000 7f143ad9d700 1 --2- 172.21.15.148:0/1249258989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f143c0f6510 0x7f143c106730 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1424013510 tx=0x7f142400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.162+0000 7f14337fe700 1 -- 172.21.15.148:0/1249258989 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1424009320 con 0x7f143c0f6510 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.162+0000 7f14337fe700 1 -- 172.21.15.148:0/1249258989 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1424013f00 con 0x7f143c0f6510 2023-12-21T03:42:45.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.162+0000 7f1441161700 1 -- 172.21.15.148:0/1249258989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143c106e20 con 0x7f143c0f6510 2023-12-21T03:42:45.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.162+0000 7f14337fe700 1 -- 172.21.15.148:0/1249258989 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1424014d90 con 0x7f143c0f6510 2023-12-21T03:42:45.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.162+0000 7f1441161700 1 -- 172.21.15.148:0/1249258989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143c1072b0 con 0x7f143c0f6510 2023-12-21T03:42:45.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.162+0000 7f14337fe700 1 -- 172.21.15.148:0/1249258989 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14240095f0 con 0x7f143c0f6510 2023-12-21T03:42:45.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.162+0000 7f1441161700 1 -- 172.21.15.148:0/1249258989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143c045ad0 con 0x7f143c0f6510 2023-12-21T03:42:45.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.163+0000 7f14337fe700 1 --2- 172.21.15.148:0/1249258989 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f142802ff60 0x7f1428032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:45.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.163+0000 7f14337fe700 1 -- 172.21.15.148:0/1249258989 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1424040fe0 con 0x7f143c0f6510 2023-12-21T03:42:45.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.163+0000 7f143a59c700 1 --2- 172.21.15.148:0/1249258989 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f142802ff60 0x7f1428032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:45.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.163+0000 7f143a59c700 1 --2- 172.21.15.148:0/1249258989 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f142802ff60 0x7f1428032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f142c0096a0 tx=0x7f142c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:45.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.167+0000 7f14337fe700 1 -- 172.21.15.148:0/1249258989 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1424014420 con 0x7f143c0f6510 2023-12-21T03:42:45.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.374+0000 7f1441161700 1 -- 172.21.15.148:0/1249258989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f143c0fbf60 con 0x7f143c0f6510 2023-12-21T03:42:45.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.375+0000 7f14337fe700 1 -- 172.21.15.148:0/1249258989 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1424010230 con 0x7f143c0f6510 2023-12-21T03:42:45.376 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:45.376 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:45.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.376+0000 7f1441161700 1 -- 172.21.15.148:0/1249258989 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f142802ff60 msgr2=0x7f1428032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:45.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.376+0000 7f1441161700 1 --2- 172.21.15.148:0/1249258989 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f142802ff60 0x7f1428032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f142c0096a0 tx=0x7f142c006b40).stop 2023-12-21T03:42:45.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.376+0000 7f1441161700 1 -- 172.21.15.148:0/1249258989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f143c0f6510 msgr2=0x7f143c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:45.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.376+0000 7f1441161700 1 --2- 172.21.15.148:0/1249258989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f143c0f6510 0x7f143c106730 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1424013510 tx=0x7f142400b040).stop 2023-12-21T03:42:45.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.376+0000 7f1441161700 1 -- 172.21.15.148:0/1249258989 shutdown_connections 2023-12-21T03:42:45.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.376+0000 7f1441161700 1 --2- 172.21.15.148:0/1249258989 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f142802ff60 0x7f1428032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:45.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.376+0000 7f1441161700 1 --2- 172.21.15.148:0/1249258989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f143c0f6510 0x7f143c106730 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:45.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.376+0000 7f1441161700 1 -- 172.21.15.148:0/1249258989 >> 172.21.15.148:0/1249258989 conn(0x7f143c0f1c70 msgr2=0x7f143c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:45.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.377+0000 7f1441161700 1 -- 172.21.15.148:0/1249258989 shutdown_connections 2023-12-21T03:42:45.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:45.377+0000 7f1441161700 1 -- 172.21.15.148:0/1249258989 wait complete. 2023-12-21T03:42:45.378 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:45.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:44.431799+0000 mgr.smithi078.zidbgq (mgr.14162) 104 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:46.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:46 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:45.375768+0000 mon.smithi078 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.148:0/1249258989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:46.782 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:46.782 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:47.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:46.432090+0000 mgr.smithi078.zidbgq (mgr.14162) 105 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:47.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.942+0000 7ff3eb59e700 1 -- 172.21.15.148:0/36792236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ec0f78e0 msgr2=0x7ff3ec0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.942+0000 7ff3eb59e700 1 --2- 172.21.15.148:0/36792236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ec0f78e0 0x7ff3ec0f7d00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7ff3dc009fb0 tx=0x7ff3dc009b30).stop 2023-12-21T03:42:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.942+0000 7ff3eb59e700 1 -- 172.21.15.148:0/36792236 shutdown_connections 2023-12-21T03:42:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.942+0000 7ff3eb59e700 1 --2- 172.21.15.148:0/36792236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ec0f78e0 0x7ff3ec0f7d00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.942+0000 7ff3eb59e700 1 -- 172.21.15.148:0/36792236 >> 172.21.15.148:0/36792236 conn(0x7ff3ec0f0120 msgr2=0x7ff3ec0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.942+0000 7ff3eb59e700 1 -- 172.21.15.148:0/36792236 shutdown_connections 2023-12-21T03:42:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.942+0000 7ff3eb59e700 1 -- 172.21.15.148:0/36792236 wait complete. 2023-12-21T03:42:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.942+0000 7ff3eb59e700 1 Processor -- start 2023-12-21T03:42:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.942+0000 7ff3eb59e700 1 -- start start 2023-12-21T03:42:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3eb59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ec0f78e0 0x7ff3ec1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3eb59e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec1069d0 con 0x7ff3ec0f78e0 2023-12-21T03:42:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3ea59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ec0f78e0 0x7ff3ec1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:47.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3ea59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ec0f78e0 0x7ff3ec1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:45372/0 (socket says 172.21.15.148:45372) 2023-12-21T03:42:47.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3ea59c700 1 -- 172.21.15.148:0/1200505831 learned_addr learned my addr 172.21.15.148:0/1200505831 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:47.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3ea59c700 1 -- 172.21.15.148:0/1200505831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc005040 con 0x7ff3ec0f78e0 2023-12-21T03:42:47.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3ea59c700 1 --2- 172.21.15.148:0/1200505831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ec0f78e0 0x7ff3ec1064e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7ff3dc013510 tx=0x7ff3dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:47.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3daffd700 1 -- 172.21.15.148:0/1200505831 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3dc009320 con 0x7ff3ec0f78e0 2023-12-21T03:42:47.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3eb59e700 1 -- 172.21.15.148:0/1200505831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec106bd0 con 0x7ff3ec0f78e0 2023-12-21T03:42:47.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3daffd700 1 -- 172.21.15.148:0/1200505831 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3dc013f00 con 0x7ff3ec0f78e0 2023-12-21T03:42:47.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3daffd700 1 -- 172.21.15.148:0/1200505831 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3dc014dc0 con 0x7ff3ec0f78e0 2023-12-21T03:42:47.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.943+0000 7ff3eb59e700 1 -- 172.21.15.148:0/1200505831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec107060 con 0x7ff3ec0f78e0 2023-12-21T03:42:47.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.944+0000 7ff3daffd700 1 -- 172.21.15.148:0/1200505831 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3dc0095f0 con 0x7ff3ec0f78e0 2023-12-21T03:42:47.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.944+0000 7ff3eb59e700 1 -- 172.21.15.148:0/1200505831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec045ad0 con 0x7ff3ec0f78e0 2023-12-21T03:42:47.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.944+0000 7ff3daffd700 1 --2- 172.21.15.148:0/1200505831 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff3d402ff60 0x7ff3d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:47.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.944+0000 7ff3daffd700 1 -- 172.21.15.148:0/1200505831 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3dc041000 con 0x7ff3ec0f78e0 2023-12-21T03:42:47.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.945+0000 7ff3e9d9b700 1 --2- 172.21.15.148:0/1200505831 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff3d402ff60 0x7ff3d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:47.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.945+0000 7ff3e9d9b700 1 --2- 172.21.15.148:0/1200505831 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff3d402ff60 0x7ff3d4032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff3e00096a0 tx=0x7ff3e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:47.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:47.949+0000 7ff3daffd700 1 -- 172.21.15.148:0/1200505831 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3dc014420 con 0x7ff3ec0f78e0 2023-12-21T03:42:48.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.155+0000 7ff3eb59e700 1 -- 172.21.15.148:0/1200505831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3ec107f10 con 0x7ff3ec0f78e0 2023-12-21T03:42:48.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.156+0000 7ff3daffd700 1 -- 172.21.15.148:0/1200505831 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff3dc010230 con 0x7ff3ec0f78e0 2023-12-21T03:42:48.157 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:48.158 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:48.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.158+0000 7ff3eb59e700 1 -- 172.21.15.148:0/1200505831 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff3d402ff60 msgr2=0x7ff3d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:48.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.158+0000 7ff3eb59e700 1 --2- 172.21.15.148:0/1200505831 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff3d402ff60 0x7ff3d4032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff3e00096a0 tx=0x7ff3e0006b40).stop 2023-12-21T03:42:48.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.158+0000 7ff3eb59e700 1 -- 172.21.15.148:0/1200505831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ec0f78e0 msgr2=0x7ff3ec1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:48.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.158+0000 7ff3eb59e700 1 --2- 172.21.15.148:0/1200505831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ec0f78e0 0x7ff3ec1064e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7ff3dc013510 tx=0x7ff3dc00b040).stop 2023-12-21T03:42:48.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.158+0000 7ff3eb59e700 1 -- 172.21.15.148:0/1200505831 shutdown_connections 2023-12-21T03:42:48.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.158+0000 7ff3eb59e700 1 --2- 172.21.15.148:0/1200505831 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff3d402ff60 0x7ff3d4032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:48.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.158+0000 7ff3eb59e700 1 --2- 172.21.15.148:0/1200505831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ec0f78e0 0x7ff3ec1064e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:48.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.158+0000 7ff3eb59e700 1 -- 172.21.15.148:0/1200505831 >> 172.21.15.148:0/1200505831 conn(0x7ff3ec0f0120 msgr2=0x7ff3ec0f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:48.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.158+0000 7ff3eb59e700 1 -- 172.21.15.148:0/1200505831 shutdown_connections 2023-12-21T03:42:48.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:48.158+0000 7ff3eb59e700 1 -- 172.21.15.148:0/1200505831 wait complete. 2023-12-21T03:42:48.160 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:48.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:48.156857+0000 mon.smithi078 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.148:0/1200505831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:49.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:48.432375+0000 mgr.smithi078.zidbgq (mgr.14162) 106 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:49.579 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:49.579 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:50.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.735+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/250472669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ba00601e0 msgr2=0x7f5ba00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.735+0000 7f5ba4cf6700 1 --2- 172.21.15.148:0/250472669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ba00601e0 0x7f5ba00f8f20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f5b94009fb0 tx=0x7f5b94009b30).stop 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.735+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/250472669 shutdown_connections 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.735+0000 7f5ba4cf6700 1 --2- 172.21.15.148:0/250472669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ba00601e0 0x7f5ba00f8f20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.735+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/250472669 >> 172.21.15.148:0/250472669 conn(0x7f5ba00f09a0 msgr2=0x7f5ba00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.735+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/250472669 shutdown_connections 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.735+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/250472669 wait complete. 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.735+0000 7f5ba4cf6700 1 Processor -- start 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.735+0000 7f5ba4cf6700 1 -- start start 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5ba4cf6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ba00601e0 0x7f5ba0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5ba4cf6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba01069f0 con 0x7f5ba00601e0 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5b9f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ba00601e0 0x7f5ba0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5b9f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ba00601e0 0x7f5ba0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:60256/0 (socket says 172.21.15.148:60256) 2023-12-21T03:42:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5b9f7fe700 1 -- 172.21.15.148:0/2683461781 learned_addr learned my addr 172.21.15.148:0/2683461781 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5b9f7fe700 1 -- 172.21.15.148:0/2683461781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b94005040 con 0x7f5ba00601e0 2023-12-21T03:42:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5b9f7fe700 1 --2- 172.21.15.148:0/2683461781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ba00601e0 0x7f5ba0106500 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5b94013510 tx=0x7f5b9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5b87fff700 1 -- 172.21.15.148:0/2683461781 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b94009320 con 0x7f5ba00601e0 2023-12-21T03:42:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5b87fff700 1 -- 172.21.15.148:0/2683461781 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b94013f00 con 0x7f5ba00601e0 2023-12-21T03:42:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/2683461781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba0106bf0 con 0x7f5ba00601e0 2023-12-21T03:42:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5b87fff700 1 -- 172.21.15.148:0/2683461781 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b94014d90 con 0x7f5ba00601e0 2023-12-21T03:42:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.736+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/2683461781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba0107080 con 0x7f5ba00601e0 2023-12-21T03:42:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.737+0000 7f5b87fff700 1 -- 172.21.15.148:0/2683461781 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b94014420 con 0x7f5ba00601e0 2023-12-21T03:42:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.737+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/2683461781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba0045ad0 con 0x7f5ba00601e0 2023-12-21T03:42:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.737+0000 7f5b87fff700 1 --2- 172.21.15.148:0/2683461781 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5b8802ff60 0x7f5b88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.737+0000 7f5b87fff700 1 -- 172.21.15.148:0/2683461781 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b94041000 con 0x7f5ba00601e0 2023-12-21T03:42:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.738+0000 7f5b9effd700 1 --2- 172.21.15.148:0/2683461781 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5b8802ff60 0x7f5b88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.738+0000 7f5b9effd700 1 --2- 172.21.15.148:0/2683461781 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5b8802ff60 0x7f5b88032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f5b900096a0 tx=0x7f5b90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.742+0000 7f5b87fff700 1 -- 172.21.15.148:0/2683461781 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b940095f0 con 0x7f5ba00601e0 2023-12-21T03:42:50.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:50.432696+0000 mgr.smithi078.zidbgq (mgr.14162) 107 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:50.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.955+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/2683461781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ba0107f30 con 0x7f5ba00601e0 2023-12-21T03:42:50.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.955+0000 7f5b87fff700 1 -- 172.21.15.148:0/2683461781 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5b9401db70 con 0x7f5ba00601e0 2023-12-21T03:42:50.957 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:50.957 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.957+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/2683461781 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5b8802ff60 msgr2=0x7f5b88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.957+0000 7f5ba4cf6700 1 --2- 172.21.15.148:0/2683461781 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5b8802ff60 0x7f5b88032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f5b900096a0 tx=0x7f5b90006b40).stop 2023-12-21T03:42:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.957+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/2683461781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ba00601e0 msgr2=0x7f5ba0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.957+0000 7f5ba4cf6700 1 --2- 172.21.15.148:0/2683461781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ba00601e0 0x7f5ba0106500 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5b94013510 tx=0x7f5b9400b040).stop 2023-12-21T03:42:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.957+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/2683461781 shutdown_connections 2023-12-21T03:42:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.957+0000 7f5ba4cf6700 1 --2- 172.21.15.148:0/2683461781 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5b8802ff60 0x7f5b88032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.957+0000 7f5ba4cf6700 1 --2- 172.21.15.148:0/2683461781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ba00601e0 0x7f5ba0106500 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.957+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/2683461781 >> 172.21.15.148:0/2683461781 conn(0x7f5ba00f09a0 msgr2=0x7f5ba00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.957+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/2683461781 shutdown_connections 2023-12-21T03:42:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:50.957+0000 7f5ba4cf6700 1 -- 172.21.15.148:0/2683461781 wait complete. 2023-12-21T03:42:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:50.956288+0000 mon.smithi078 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.148:0/2683461781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:52.364 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:52.364 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:52.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:52.433016+0000 mgr.smithi078.zidbgq (mgr.14162) 108 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:53.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.558+0000 7fcc722c0700 1 -- 172.21.15.148:0/3220969668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc6c0601c0 msgr2=0x7fcc6c0f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:53.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.558+0000 7fcc722c0700 1 --2- 172.21.15.148:0/3220969668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc6c0601c0 0x7fcc6c0f91b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fcc5c009fb0 tx=0x7fcc5c009b30).stop 2023-12-21T03:42:53.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.558+0000 7fcc722c0700 1 -- 172.21.15.148:0/3220969668 shutdown_connections 2023-12-21T03:42:53.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.558+0000 7fcc722c0700 1 --2- 172.21.15.148:0/3220969668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc6c0601c0 0x7fcc6c0f91b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:53.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.558+0000 7fcc722c0700 1 -- 172.21.15.148:0/3220969668 >> 172.21.15.148:0/3220969668 conn(0x7fcc6c0f0b70 msgr2=0x7fcc6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:53.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.558+0000 7fcc722c0700 1 -- 172.21.15.148:0/3220969668 shutdown_connections 2023-12-21T03:42:53.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.558+0000 7fcc722c0700 1 -- 172.21.15.148:0/3220969668 wait complete. 2023-12-21T03:42:53.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.558+0000 7fcc722c0700 1 Processor -- start 2023-12-21T03:42:53.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.559+0000 7fcc722c0700 1 -- start start 2023-12-21T03:42:53.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.559+0000 7fcc722c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc6c0601c0 0x7fcc6c0f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:53.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.559+0000 7fcc722c0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c0f50b0 con 0x7fcc6c0601c0 2023-12-21T03:42:53.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.559+0000 7fcc6bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc6c0601c0 0x7fcc6c0f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:53.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.559+0000 7fcc6bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc6c0601c0 0x7fcc6c0f6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:60280/0 (socket says 172.21.15.148:60280) 2023-12-21T03:42:53.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.559+0000 7fcc6bfff700 1 -- 172.21.15.148:0/1079654373 learned_addr learned my addr 172.21.15.148:0/1079654373 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:53.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.559+0000 7fcc6bfff700 1 -- 172.21.15.148:0/1079654373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c005040 con 0x7fcc6c0601c0 2023-12-21T03:42:53.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.559+0000 7fcc6bfff700 1 --2- 172.21.15.148:0/1079654373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc6c0601c0 0x7fcc6c0f6c30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fcc5c009830 tx=0x7fcc5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:53.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.560+0000 7fcc68ff9700 1 -- 172.21.15.148:0/1079654373 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc5c009320 con 0x7fcc6c0601c0 2023-12-21T03:42:53.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.560+0000 7fcc68ff9700 1 -- 172.21.15.148:0/1079654373 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc5c0136c0 con 0x7fcc6c0601c0 2023-12-21T03:42:53.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.560+0000 7fcc68ff9700 1 -- 172.21.15.148:0/1079654373 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc5c01ee90 con 0x7fcc6c0601c0 2023-12-21T03:42:53.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.560+0000 7fcc722c0700 1 -- 172.21.15.148:0/1079654373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6c0f5250 con 0x7fcc6c0601c0 2023-12-21T03:42:53.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.560+0000 7fcc722c0700 1 -- 172.21.15.148:0/1079654373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6c0f56e0 con 0x7fcc6c0601c0 2023-12-21T03:42:53.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.560+0000 7fcc68ff9700 1 -- 172.21.15.148:0/1079654373 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc5c01e420 con 0x7fcc6c0601c0 2023-12-21T03:42:53.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.560+0000 7fcc722c0700 1 -- 172.21.15.148:0/1079654373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6c059100 con 0x7fcc6c0601c0 2023-12-21T03:42:53.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.561+0000 7fcc68ff9700 1 --2- 172.21.15.148:0/1079654373 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcc5402ff10 0x7fcc540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:53.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.561+0000 7fcc68ff9700 1 -- 172.21.15.148:0/1079654373 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc5c0410b0 con 0x7fcc6c0601c0 2023-12-21T03:42:53.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.561+0000 7fcc6b7fe700 1 --2- 172.21.15.148:0/1079654373 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcc5402ff10 0x7fcc540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:53.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.561+0000 7fcc6b7fe700 1 --2- 172.21.15.148:0/1079654373 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcc5402ff10 0x7fcc540323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fcc600096a0 tx=0x7fcc60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:53.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.565+0000 7fcc68ff9700 1 -- 172.21.15.148:0/1079654373 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc5c06d2b0 con 0x7fcc6c0601c0 2023-12-21T03:42:53.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.769+0000 7fcc722c0700 1 -- 172.21.15.148:0/1079654373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc6c0fbf50 con 0x7fcc6c0601c0 2023-12-21T03:42:53.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.770+0000 7fcc68ff9700 1 -- 172.21.15.148:0/1079654373 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc5c02c2d0 con 0x7fcc6c0601c0 2023-12-21T03:42:53.771 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:53.772 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:53.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.771+0000 7fcc722c0700 1 -- 172.21.15.148:0/1079654373 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcc5402ff10 msgr2=0x7fcc540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:53.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.771+0000 7fcc722c0700 1 --2- 172.21.15.148:0/1079654373 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcc5402ff10 0x7fcc540323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fcc600096a0 tx=0x7fcc60006b40).stop 2023-12-21T03:42:53.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.771+0000 7fcc722c0700 1 -- 172.21.15.148:0/1079654373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc6c0601c0 msgr2=0x7fcc6c0f6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:53.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.771+0000 7fcc722c0700 1 --2- 172.21.15.148:0/1079654373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc6c0601c0 0x7fcc6c0f6c30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fcc5c009830 tx=0x7fcc5c00b040).stop 2023-12-21T03:42:53.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.771+0000 7fcc722c0700 1 -- 172.21.15.148:0/1079654373 shutdown_connections 2023-12-21T03:42:53.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.771+0000 7fcc722c0700 1 --2- 172.21.15.148:0/1079654373 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcc5402ff10 0x7fcc540323d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:53.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.771+0000 7fcc722c0700 1 --2- 172.21.15.148:0/1079654373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc6c0601c0 0x7fcc6c0f6c30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:53.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.771+0000 7fcc722c0700 1 -- 172.21.15.148:0/1079654373 >> 172.21.15.148:0/1079654373 conn(0x7fcc6c0f0b70 msgr2=0x7fcc6c0f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:53.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.772+0000 7fcc722c0700 1 -- 172.21.15.148:0/1079654373 shutdown_connections 2023-12-21T03:42:53.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:53.772+0000 7fcc722c0700 1 -- 172.21.15.148:0/1079654373 wait complete. 2023-12-21T03:42:53.774 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:54.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:53.770790+0000 mon.smithi078 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.148:0/1079654373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:55.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:55.144 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:55.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:54.433423+0000 mgr.smithi078.zidbgq (mgr.14162) 109 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:56.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.327+0000 7f66affff700 1 -- 172.21.15.148:0/3695052883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00f62a0 msgr2=0x7f66b00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:56.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.327+0000 7f66affff700 1 --2- 172.21.15.148:0/3695052883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00f62a0 0x7f66b00f66c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f66a0009fb0 tx=0x7f66a0009b30).stop 2023-12-21T03:42:56.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.327+0000 7f66affff700 1 -- 172.21.15.148:0/3695052883 shutdown_connections 2023-12-21T03:42:56.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.327+0000 7f66affff700 1 --2- 172.21.15.148:0/3695052883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00f62a0 0x7f66b00f66c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:56.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.327+0000 7f66affff700 1 -- 172.21.15.148:0/3695052883 >> 172.21.15.148:0/3695052883 conn(0x7f66b00f1ac0 msgr2=0x7f66b00f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.327+0000 7f66affff700 1 -- 172.21.15.148:0/3695052883 shutdown_connections 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.327+0000 7f66affff700 1 -- 172.21.15.148:0/3695052883 wait complete. 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.327+0000 7f66affff700 1 Processor -- start 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.328+0000 7f66affff700 1 -- start start 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.328+0000 7f66affff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00f62a0 0x7f66b01064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.328+0000 7f66affff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b0106990 con 0x7f66b00f62a0 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.328+0000 7f66aeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00f62a0 0x7f66b01064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.328+0000 7f66aeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00f62a0 0x7f66b01064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:60304/0 (socket says 172.21.15.148:60304) 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.328+0000 7f66aeffd700 1 -- 172.21.15.148:0/353218390 learned_addr learned my addr 172.21.15.148:0/353218390 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.328+0000 7f66aeffd700 1 -- 172.21.15.148:0/353218390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a0005040 con 0x7f66b00f62a0 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.328+0000 7f66aeffd700 1 --2- 172.21.15.148:0/353218390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00f62a0 0x7f66b01064a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f66a0013510 tx=0x7f66a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:56.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.328+0000 7f66977fe700 1 -- 172.21.15.148:0/353218390 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66a0009320 con 0x7f66b00f62a0 2023-12-21T03:42:56.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.329+0000 7f66977fe700 1 -- 172.21.15.148:0/353218390 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66a0013f00 con 0x7f66b00f62a0 2023-12-21T03:42:56.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.329+0000 7f66affff700 1 -- 172.21.15.148:0/353218390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b0106b90 con 0x7f66b00f62a0 2023-12-21T03:42:56.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.329+0000 7f66977fe700 1 -- 172.21.15.148:0/353218390 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66a0014dc0 con 0x7f66b00f62a0 2023-12-21T03:42:56.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.329+0000 7f66affff700 1 -- 172.21.15.148:0/353218390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b0107020 con 0x7f66b00f62a0 2023-12-21T03:42:56.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.329+0000 7f66977fe700 1 -- 172.21.15.148:0/353218390 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66a00095f0 con 0x7f66b00f62a0 2023-12-21T03:42:56.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.329+0000 7f66affff700 1 -- 172.21.15.148:0/353218390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b0045ad0 con 0x7f66b00f62a0 2023-12-21T03:42:56.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.330+0000 7f66977fe700 1 --2- 172.21.15.148:0/353218390 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f669802ff60 0x7f6698032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:56.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.330+0000 7f66977fe700 1 -- 172.21.15.148:0/353218390 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66a0041000 con 0x7f66b00f62a0 2023-12-21T03:42:56.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.330+0000 7f66ae7fc700 1 --2- 172.21.15.148:0/353218390 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f669802ff60 0x7f6698032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:56.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.330+0000 7f66ae7fc700 1 --2- 172.21.15.148:0/353218390 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f669802ff60 0x7f6698032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f66a40096a0 tx=0x7f66a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:56.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.334+0000 7f66977fe700 1 -- 172.21.15.148:0/353218390 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66a001c790 con 0x7f66b00f62a0 2023-12-21T03:42:56.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.541+0000 7f66affff700 1 -- 172.21.15.148:0/353218390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66b0107ed0 con 0x7f66b00f62a0 2023-12-21T03:42:56.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.542+0000 7f66977fe700 1 -- 172.21.15.148:0/353218390 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f66a0010230 con 0x7f66b00f62a0 2023-12-21T03:42:56.543 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:56.543 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:56.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.543+0000 7f66affff700 1 -- 172.21.15.148:0/353218390 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f669802ff60 msgr2=0x7f6698032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.543+0000 7f66affff700 1 --2- 172.21.15.148:0/353218390 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f669802ff60 0x7f6698032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f66a40096a0 tx=0x7f66a4006b40).stop 2023-12-21T03:42:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.543+0000 7f66affff700 1 -- 172.21.15.148:0/353218390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00f62a0 msgr2=0x7f66b01064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.543+0000 7f66affff700 1 --2- 172.21.15.148:0/353218390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00f62a0 0x7f66b01064a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f66a0013510 tx=0x7f66a000b040).stop 2023-12-21T03:42:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.543+0000 7f66affff700 1 -- 172.21.15.148:0/353218390 shutdown_connections 2023-12-21T03:42:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.543+0000 7f66affff700 1 --2- 172.21.15.148:0/353218390 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f669802ff60 0x7f6698032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.543+0000 7f66affff700 1 --2- 172.21.15.148:0/353218390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00f62a0 0x7f66b01064a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.543+0000 7f66affff700 1 -- 172.21.15.148:0/353218390 >> 172.21.15.148:0/353218390 conn(0x7f66b00f1ac0 msgr2=0x7f66b00f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.543+0000 7f66affff700 1 -- 172.21.15.148:0/353218390 shutdown_connections 2023-12-21T03:42:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:56.544+0000 7f66affff700 1 -- 172.21.15.148:0/353218390 wait complete. 2023-12-21T03:42:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:42:56.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:56.433695+0000 mgr.smithi078.zidbgq (mgr.14162) 110 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:56.542664+0000 mon.smithi078 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.148:0/353218390' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:42:57.915 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:42:57.915 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:42:58.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:42:58 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:42:58.433934+0000 mgr.smithi078.zidbgq (mgr.14162) 111 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:42:59.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.040+0000 7f83067b3700 1 -- 172.21.15.148:0/3696858447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83000f5410 msgr2=0x7f83000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:59.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.040+0000 7f83067b3700 1 --2- 172.21.15.148:0/3696858447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83000f5410 0x7f83000f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f82f0009fb0 tx=0x7f82f0009b30).stop 2023-12-21T03:42:59.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.040+0000 7f83067b3700 1 -- 172.21.15.148:0/3696858447 shutdown_connections 2023-12-21T03:42:59.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.040+0000 7f83067b3700 1 --2- 172.21.15.148:0/3696858447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83000f5410 0x7f83000f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:59.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.040+0000 7f83067b3700 1 -- 172.21.15.148:0/3696858447 >> 172.21.15.148:0/3696858447 conn(0x7f83000f0b70 msgr2=0x7f83000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:59.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.040+0000 7f83067b3700 1 -- 172.21.15.148:0/3696858447 shutdown_connections 2023-12-21T03:42:59.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.040+0000 7f83067b3700 1 -- 172.21.15.148:0/3696858447 wait complete. 2023-12-21T03:42:59.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.041+0000 7f83067b3700 1 Processor -- start 2023-12-21T03:42:59.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.041+0000 7f83067b3700 1 -- start start 2023-12-21T03:42:59.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.041+0000 7f83067b3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83000f5410 0x7f8300109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:59.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.041+0000 7f83067b3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8300060220 con 0x7f83000f5410 2023-12-21T03:42:59.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.041+0000 7f82fffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83000f5410 0x7f8300109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:59.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.041+0000 7f82fffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83000f5410 0x7f8300109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:60326/0 (socket says 172.21.15.148:60326) 2023-12-21T03:42:59.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.041+0000 7f82fffff700 1 -- 172.21.15.148:0/3576908714 learned_addr learned my addr 172.21.15.148:0/3576908714 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:42:59.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.042+0000 7f82fffff700 1 -- 172.21.15.148:0/3576908714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f0005040 con 0x7f83000f5410 2023-12-21T03:42:59.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.042+0000 7f82fffff700 1 --2- 172.21.15.148:0/3576908714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83000f5410 0x7f8300109450 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f82f0013510 tx=0x7f82f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:59.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.042+0000 7f82fcff9700 1 -- 172.21.15.148:0/3576908714 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82f0009320 con 0x7f83000f5410 2023-12-21T03:42:59.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.042+0000 7f83067b3700 1 -- 172.21.15.148:0/3576908714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8300060420 con 0x7f83000f5410 2023-12-21T03:42:59.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.042+0000 7f82fcff9700 1 -- 172.21.15.148:0/3576908714 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82f0013f00 con 0x7f83000f5410 2023-12-21T03:42:59.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.042+0000 7f82fcff9700 1 -- 172.21.15.148:0/3576908714 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82f0014d90 con 0x7f83000f5410 2023-12-21T03:42:59.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.042+0000 7f83067b3700 1 -- 172.21.15.148:0/3576908714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8300109b00 con 0x7f83000f5410 2023-12-21T03:42:59.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.043+0000 7f82fcff9700 1 -- 172.21.15.148:0/3576908714 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82f0014420 con 0x7f83000f5410 2023-12-21T03:42:59.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.043+0000 7f83067b3700 1 -- 172.21.15.148:0/3576908714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8300059100 con 0x7f83000f5410 2023-12-21T03:42:59.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.043+0000 7f82fcff9700 1 --2- 172.21.15.148:0/3576908714 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f82ec02ff60 0x7f82ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:42:59.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.043+0000 7f82fcff9700 1 -- 172.21.15.148:0/3576908714 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82f0041000 con 0x7f83000f5410 2023-12-21T03:42:59.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.043+0000 7f82ff7fe700 1 --2- 172.21.15.148:0/3576908714 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f82ec02ff60 0x7f82ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:42:59.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.044+0000 7f82ff7fe700 1 --2- 172.21.15.148:0/3576908714 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f82ec02ff60 0x7f82ec032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f82e80096a0 tx=0x7f82e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:42:59.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.048+0000 7f82fcff9700 1 -- 172.21.15.148:0/3576908714 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82f00095f0 con 0x7f83000f5410 2023-12-21T03:42:59.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.254+0000 7f83067b3700 1 -- 172.21.15.148:0/3576908714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8300045ad0 con 0x7f83000f5410 2023-12-21T03:42:59.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.255+0000 7f82fcff9700 1 -- 172.21.15.148:0/3576908714 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f82f001db70 con 0x7f83000f5410 2023-12-21T03:42:59.256 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:42:59.256 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:42:59.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.256+0000 7f83067b3700 1 -- 172.21.15.148:0/3576908714 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f82ec02ff60 msgr2=0x7f82ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:59.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.256+0000 7f83067b3700 1 --2- 172.21.15.148:0/3576908714 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f82ec02ff60 0x7f82ec032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f82e80096a0 tx=0x7f82e8006b40).stop 2023-12-21T03:42:59.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.256+0000 7f83067b3700 1 -- 172.21.15.148:0/3576908714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83000f5410 msgr2=0x7f8300109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:42:59.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.256+0000 7f83067b3700 1 --2- 172.21.15.148:0/3576908714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83000f5410 0x7f8300109450 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f82f0013510 tx=0x7f82f000b040).stop 2023-12-21T03:42:59.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.256+0000 7f83067b3700 1 -- 172.21.15.148:0/3576908714 shutdown_connections 2023-12-21T03:42:59.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.256+0000 7f83067b3700 1 --2- 172.21.15.148:0/3576908714 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f82ec02ff60 0x7f82ec032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:59.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.256+0000 7f83067b3700 1 --2- 172.21.15.148:0/3576908714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83000f5410 0x7f8300109450 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:42:59.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.256+0000 7f83067b3700 1 -- 172.21.15.148:0/3576908714 >> 172.21.15.148:0/3576908714 conn(0x7f83000f0b70 msgr2=0x7f83000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:42:59.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.256+0000 7f83067b3700 1 -- 172.21.15.148:0/3576908714 shutdown_connections 2023-12-21T03:42:59.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:42:59.256+0000 7f83067b3700 1 -- 172.21.15.148:0/3576908714 wait complete. 2023-12-21T03:42:59.258 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:00.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:42:59.255549+0000 mon.smithi078 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.148:0/3576908714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:00.639 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:00.639 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:01.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:00.434344+0000 mgr.smithi078.zidbgq (mgr.14162) 112 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:01.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.733+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/2475792323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7b40f5410 msgr2=0x7fd7b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:01.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.733+0000 7fd7bc2d6700 1 --2- 172.21.15.148:0/2475792323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7b40f5410 0x7fd7b40f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fd7b0009fb0 tx=0x7fd7b0009b30).stop 2023-12-21T03:43:01.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.733+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/2475792323 shutdown_connections 2023-12-21T03:43:01.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.733+0000 7fd7bc2d6700 1 --2- 172.21.15.148:0/2475792323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7b40f5410 0x7fd7b40f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:01.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.733+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/2475792323 >> 172.21.15.148:0/2475792323 conn(0x7fd7b40f0b70 msgr2=0x7fd7b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:01.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.733+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/2475792323 shutdown_connections 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.733+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/2475792323 wait complete. 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.734+0000 7fd7bc2d6700 1 Processor -- start 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.734+0000 7fd7bc2d6700 1 -- start start 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.734+0000 7fd7bc2d6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7b40f5410 0x7fd7b4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.734+0000 7fd7bc2d6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b4105b20 con 0x7fd7b40f5410 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.734+0000 7fd7ba072700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7b40f5410 0x7fd7b4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.734+0000 7fd7ba072700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7b40f5410 0x7fd7b4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:51830/0 (socket says 172.21.15.148:51830) 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.734+0000 7fd7ba072700 1 -- 172.21.15.148:0/804818550 learned_addr learned my addr 172.21.15.148:0/804818550 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.734+0000 7fd7ba072700 1 -- 172.21.15.148:0/804818550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b0005040 con 0x7fd7b40f5410 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.734+0000 7fd7ba072700 1 --2- 172.21.15.148:0/804818550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7b40f5410 0x7fd7b4105630 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fd7b0013510 tx=0x7fd7b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.734+0000 7fd7aaffd700 1 -- 172.21.15.148:0/804818550 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7b0009320 con 0x7fd7b40f5410 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.735+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/804818550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b4105d20 con 0x7fd7b40f5410 2023-12-21T03:43:01.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.735+0000 7fd7aaffd700 1 -- 172.21.15.148:0/804818550 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7b0013f00 con 0x7fd7b40f5410 2023-12-21T03:43:01.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.735+0000 7fd7aaffd700 1 -- 172.21.15.148:0/804818550 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7b0014d90 con 0x7fd7b40f5410 2023-12-21T03:43:01.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.735+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/804818550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b41061b0 con 0x7fd7b40f5410 2023-12-21T03:43:01.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.736+0000 7fd7aaffd700 1 -- 172.21.15.148:0/804818550 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7b0014420 con 0x7fd7b40f5410 2023-12-21T03:43:01.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.736+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/804818550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b4045ad0 con 0x7fd7b40f5410 2023-12-21T03:43:01.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.736+0000 7fd7aaffd700 1 --2- 172.21.15.148:0/804818550 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd7a002ff60 0x7fd7a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:01.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.736+0000 7fd7aaffd700 1 -- 172.21.15.148:0/804818550 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd7b0041000 con 0x7fd7b40f5410 2023-12-21T03:43:01.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.736+0000 7fd7b9871700 1 --2- 172.21.15.148:0/804818550 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd7a002ff60 0x7fd7a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:01.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.737+0000 7fd7b9871700 1 --2- 172.21.15.148:0/804818550 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd7a002ff60 0x7fd7a0032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fd7a40096a0 tx=0x7fd7a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:01.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.741+0000 7fd7aaffd700 1 -- 172.21.15.148:0/804818550 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7b00095f0 con 0x7fd7b40f5410 2023-12-21T03:43:01.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.948+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/804818550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7b40fae60 con 0x7fd7b40f5410 2023-12-21T03:43:01.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.948+0000 7fd7aaffd700 1 -- 172.21.15.148:0/804818550 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd7b001db70 con 0x7fd7b40f5410 2023-12-21T03:43:01.950 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:01.950 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:01.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.949+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/804818550 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd7a002ff60 msgr2=0x7fd7a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:01.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.950+0000 7fd7bc2d6700 1 --2- 172.21.15.148:0/804818550 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd7a002ff60 0x7fd7a0032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fd7a40096a0 tx=0x7fd7a4006b40).stop 2023-12-21T03:43:01.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.950+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/804818550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7b40f5410 msgr2=0x7fd7b4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:01.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.950+0000 7fd7bc2d6700 1 --2- 172.21.15.148:0/804818550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7b40f5410 0x7fd7b4105630 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fd7b0013510 tx=0x7fd7b000b040).stop 2023-12-21T03:43:01.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.950+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/804818550 shutdown_connections 2023-12-21T03:43:01.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.950+0000 7fd7bc2d6700 1 --2- 172.21.15.148:0/804818550 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd7a002ff60 0x7fd7a0032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:01.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.950+0000 7fd7bc2d6700 1 --2- 172.21.15.148:0/804818550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7b40f5410 0x7fd7b4105630 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:01.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.950+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/804818550 >> 172.21.15.148:0/804818550 conn(0x7fd7b40f0b70 msgr2=0x7fd7b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:01.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.950+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/804818550 shutdown_connections 2023-12-21T03:43:01.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:01.950+0000 7fd7bc2d6700 1 -- 172.21.15.148:0/804818550 wait complete. 2023-12-21T03:43:01.953 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:02.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:01.949869+0000 mon.smithi078 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.148:0/804818550' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:03.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:03.363 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:03.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:02.434692+0000 mgr.smithi078.zidbgq (mgr.14162) 113 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:04.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.517+0000 7f040083d700 1 -- 172.21.15.148:0/3961755803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc0f5410 msgr2=0x7f03fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:04.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.517+0000 7f040083d700 1 --2- 172.21.15.148:0/3961755803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc0f5410 0x7f03fc0f5830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f03ec009fb0 tx=0x7f03ec009b30).stop 2023-12-21T03:43:04.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.517+0000 7f040083d700 1 -- 172.21.15.148:0/3961755803 shutdown_connections 2023-12-21T03:43:04.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.517+0000 7f040083d700 1 --2- 172.21.15.148:0/3961755803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc0f5410 0x7f03fc0f5830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:04.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.517+0000 7f040083d700 1 -- 172.21.15.148:0/3961755803 >> 172.21.15.148:0/3961755803 conn(0x7f03fc0f0b70 msgr2=0x7f03fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:04.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.517+0000 7f040083d700 1 -- 172.21.15.148:0/3961755803 shutdown_connections 2023-12-21T03:43:04.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.517+0000 7f040083d700 1 -- 172.21.15.148:0/3961755803 wait complete. 2023-12-21T03:43:04.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.517+0000 7f040083d700 1 Processor -- start 2023-12-21T03:43:04.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.517+0000 7f040083d700 1 -- start start 2023-12-21T03:43:04.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f040083d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc0f5410 0x7f03fc105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f040083d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc105af0 con 0x7f03fc0f5410 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f03fa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc0f5410 0x7f03fc105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f03fa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc0f5410 0x7f03fc105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:51846/0 (socket says 172.21.15.148:51846) 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f03fa59c700 1 -- 172.21.15.148:0/3591020108 learned_addr learned my addr 172.21.15.148:0/3591020108 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f03fa59c700 1 -- 172.21.15.148:0/3591020108 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec005040 con 0x7f03fc0f5410 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f03fa59c700 1 --2- 172.21.15.148:0/3591020108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc0f5410 0x7f03fc105600 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f03ec013510 tx=0x7f03ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f03eaffd700 1 -- 172.21.15.148:0/3591020108 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03ec009320 con 0x7f03fc0f5410 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f040083d700 1 -- 172.21.15.148:0/3591020108 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03fc105cf0 con 0x7f03fc0f5410 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f03eaffd700 1 -- 172.21.15.148:0/3591020108 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03ec013f00 con 0x7f03fc0f5410 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f03eaffd700 1 -- 172.21.15.148:0/3591020108 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03ec014d90 con 0x7f03fc0f5410 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.518+0000 7f040083d700 1 -- 172.21.15.148:0/3591020108 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03fc106180 con 0x7f03fc0f5410 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.519+0000 7f03eaffd700 1 -- 172.21.15.148:0/3591020108 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03ec0095f0 con 0x7f03fc0f5410 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.519+0000 7f040083d700 1 -- 172.21.15.148:0/3591020108 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03fc045ad0 con 0x7f03fc0f5410 2023-12-21T03:43:04.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.519+0000 7f03eaffd700 1 --2- 172.21.15.148:0/3591020108 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03e402ff60 0x7f03e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:04.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.519+0000 7f03eaffd700 1 -- 172.21.15.148:0/3591020108 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f03ec0410a0 con 0x7f03fc0f5410 2023-12-21T03:43:04.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.520+0000 7f03f9d9b700 1 --2- 172.21.15.148:0/3591020108 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03e402ff60 0x7f03e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:04.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.520+0000 7f03f9d9b700 1 --2- 172.21.15.148:0/3591020108 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03e402ff60 0x7f03e4032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f03f00096a0 tx=0x7f03f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:04.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.524+0000 7f03eaffd700 1 -- 172.21.15.148:0/3591020108 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03ec01d7f0 con 0x7f03fc0f5410 2023-12-21T03:43:04.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.730+0000 7f040083d700 1 -- 172.21.15.148:0/3591020108 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03fc0fae60 con 0x7f03fc0f5410 2023-12-21T03:43:04.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.731+0000 7f03eaffd700 1 -- 172.21.15.148:0/3591020108 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f03ec010230 con 0x7f03fc0f5410 2023-12-21T03:43:04.733 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:04.733 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:04.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.732+0000 7f040083d700 1 -- 172.21.15.148:0/3591020108 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03e402ff60 msgr2=0x7f03e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:04.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.732+0000 7f040083d700 1 --2- 172.21.15.148:0/3591020108 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03e402ff60 0x7f03e4032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f03f00096a0 tx=0x7f03f0006b40).stop 2023-12-21T03:43:04.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.732+0000 7f040083d700 1 -- 172.21.15.148:0/3591020108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc0f5410 msgr2=0x7f03fc105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:04.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.732+0000 7f040083d700 1 --2- 172.21.15.148:0/3591020108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc0f5410 0x7f03fc105600 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f03ec013510 tx=0x7f03ec00b040).stop 2023-12-21T03:43:04.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.732+0000 7f040083d700 1 -- 172.21.15.148:0/3591020108 shutdown_connections 2023-12-21T03:43:04.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.732+0000 7f040083d700 1 --2- 172.21.15.148:0/3591020108 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03e402ff60 0x7f03e4032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:04.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.732+0000 7f040083d700 1 --2- 172.21.15.148:0/3591020108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc0f5410 0x7f03fc105600 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:04.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.732+0000 7f040083d700 1 -- 172.21.15.148:0/3591020108 >> 172.21.15.148:0/3591020108 conn(0x7f03fc0f0b70 msgr2=0x7f03fc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:04.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.732+0000 7f040083d700 1 -- 172.21.15.148:0/3591020108 shutdown_connections 2023-12-21T03:43:04.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:04.732+0000 7f040083d700 1 -- 172.21.15.148:0/3591020108 wait complete. 2023-12-21T03:43:04.736 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:05.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:04.435014+0000 mgr.smithi078.zidbgq (mgr.14162) 114 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:05.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:04.732690+0000 mon.smithi078 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.148:0/3591020108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:06.067 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:06.067 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:06.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:06 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:06.435422+0000 mgr.smithi078.zidbgq (mgr.14162) 115 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:07.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.133+0000 7f57f7416700 1 -- 172.21.15.148:0/1172858721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f00f6510 msgr2=0x7f57f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:07.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.133+0000 7f57f7416700 1 --2- 172.21.15.148:0/1172858721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f00f6510 0x7f57f00f6930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f57e0009fb0 tx=0x7f57e0009b30).stop 2023-12-21T03:43:07.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.133+0000 7f57f7416700 1 -- 172.21.15.148:0/1172858721 shutdown_connections 2023-12-21T03:43:07.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.133+0000 7f57f7416700 1 --2- 172.21.15.148:0/1172858721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f00f6510 0x7f57f00f6930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:07.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.133+0000 7f57f7416700 1 -- 172.21.15.148:0/1172858721 >> 172.21.15.148:0/1172858721 conn(0x7f57f00f1c70 msgr2=0x7f57f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:07.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.133+0000 7f57f7416700 1 -- 172.21.15.148:0/1172858721 shutdown_connections 2023-12-21T03:43:07.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.133+0000 7f57f7416700 1 -- 172.21.15.148:0/1172858721 wait complete. 2023-12-21T03:43:07.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.134+0000 7f57f7416700 1 Processor -- start 2023-12-21T03:43:07.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.134+0000 7f57f7416700 1 -- start start 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.134+0000 7f57f7416700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f00f6510 0x7f57f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.134+0000 7f57f7416700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f0106c20 con 0x7f57f00f6510 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.134+0000 7f57f51b2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f00f6510 0x7f57f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.134+0000 7f57f51b2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f00f6510 0x7f57f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:51872/0 (socket says 172.21.15.148:51872) 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.134+0000 7f57f51b2700 1 -- 172.21.15.148:0/460434165 learned_addr learned my addr 172.21.15.148:0/460434165 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.134+0000 7f57f51b2700 1 -- 172.21.15.148:0/460434165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e0005040 con 0x7f57f00f6510 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.135+0000 7f57f51b2700 1 --2- 172.21.15.148:0/460434165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f00f6510 0x7f57f0106730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f57e0013510 tx=0x7f57e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.135+0000 7f57e5ffb700 1 -- 172.21.15.148:0/460434165 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57e0009320 con 0x7f57f00f6510 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.135+0000 7f57e5ffb700 1 -- 172.21.15.148:0/460434165 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57e0013f00 con 0x7f57f00f6510 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.135+0000 7f57f7416700 1 -- 172.21.15.148:0/460434165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f0106e20 con 0x7f57f00f6510 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.135+0000 7f57e5ffb700 1 -- 172.21.15.148:0/460434165 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57e0014d90 con 0x7f57f00f6510 2023-12-21T03:43:07.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.135+0000 7f57f7416700 1 -- 172.21.15.148:0/460434165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f01072b0 con 0x7f57f00f6510 2023-12-21T03:43:07.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.136+0000 7f57e5ffb700 1 -- 172.21.15.148:0/460434165 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57e00095f0 con 0x7f57f00f6510 2023-12-21T03:43:07.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.136+0000 7f57f7416700 1 -- 172.21.15.148:0/460434165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f0045ad0 con 0x7f57f00f6510 2023-12-21T03:43:07.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.136+0000 7f57e5ffb700 1 --2- 172.21.15.148:0/460434165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57dc02ff60 0x7f57dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:07.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.136+0000 7f57e5ffb700 1 -- 172.21.15.148:0/460434165 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57e00410a0 con 0x7f57f00f6510 2023-12-21T03:43:07.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.136+0000 7f57f49b1700 1 --2- 172.21.15.148:0/460434165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57dc02ff60 0x7f57dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:07.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.137+0000 7f57f49b1700 1 --2- 172.21.15.148:0/460434165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57dc02ff60 0x7f57dc032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f57ec0096a0 tx=0x7f57ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:07.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.141+0000 7f57e5ffb700 1 -- 172.21.15.148:0/460434165 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57e001d7f0 con 0x7f57f00f6510 2023-12-21T03:43:07.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.347+0000 7f57f7416700 1 -- 172.21.15.148:0/460434165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57f00fbf60 con 0x7f57f00f6510 2023-12-21T03:43:07.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.348+0000 7f57e5ffb700 1 -- 172.21.15.148:0/460434165 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f57e0010230 con 0x7f57f00f6510 2023-12-21T03:43:07.350 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:07.350 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:07.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.350+0000 7f57f7416700 1 -- 172.21.15.148:0/460434165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57dc02ff60 msgr2=0x7f57dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:07.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.350+0000 7f57f7416700 1 --2- 172.21.15.148:0/460434165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57dc02ff60 0x7f57dc032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f57ec0096a0 tx=0x7f57ec006b40).stop 2023-12-21T03:43:07.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.350+0000 7f57f7416700 1 -- 172.21.15.148:0/460434165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f00f6510 msgr2=0x7f57f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:07.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.350+0000 7f57f7416700 1 --2- 172.21.15.148:0/460434165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f00f6510 0x7f57f0106730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f57e0013510 tx=0x7f57e000b040).stop 2023-12-21T03:43:07.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.350+0000 7f57f7416700 1 -- 172.21.15.148:0/460434165 shutdown_connections 2023-12-21T03:43:07.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.350+0000 7f57f7416700 1 --2- 172.21.15.148:0/460434165 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57dc02ff60 0x7f57dc032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:07.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.350+0000 7f57f7416700 1 --2- 172.21.15.148:0/460434165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f00f6510 0x7f57f0106730 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:07.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.350+0000 7f57f7416700 1 -- 172.21.15.148:0/460434165 >> 172.21.15.148:0/460434165 conn(0x7f57f00f1c70 msgr2=0x7f57f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:07.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.350+0000 7f57f7416700 1 -- 172.21.15.148:0/460434165 shutdown_connections 2023-12-21T03:43:07.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:07.350+0000 7f57f7416700 1 -- 172.21.15.148:0/460434165 wait complete. 2023-12-21T03:43:07.353 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:07.349765+0000 mon.smithi078 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.148:0/460434165' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:08.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:08.747 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:08.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:08.435768+0000 mgr.smithi078.zidbgq (mgr.14162) 116 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.881+0000 7f2bc711f700 1 -- 172.21.15.148:0/2217183299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc00f5410 msgr2=0x7f2bc00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.881+0000 7f2bc711f700 1 --2- 172.21.15.148:0/2217183299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc00f5410 0x7f2bc00f5830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f2bb4009fb0 tx=0x7f2bb4009b30).stop 2023-12-21T03:43:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.881+0000 7f2bc711f700 1 -- 172.21.15.148:0/2217183299 shutdown_connections 2023-12-21T03:43:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.881+0000 7f2bc711f700 1 --2- 172.21.15.148:0/2217183299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc00f5410 0x7f2bc00f5830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.881+0000 7f2bc711f700 1 -- 172.21.15.148:0/2217183299 >> 172.21.15.148:0/2217183299 conn(0x7f2bc00f0b70 msgr2=0x7f2bc00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.881+0000 7f2bc711f700 1 -- 172.21.15.148:0/2217183299 shutdown_connections 2023-12-21T03:43:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.881+0000 7f2bc711f700 1 -- 172.21.15.148:0/2217183299 wait complete. 2023-12-21T03:43:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.882+0000 7f2bc711f700 1 Processor -- start 2023-12-21T03:43:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.882+0000 7f2bc711f700 1 -- start start 2023-12-21T03:43:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.882+0000 7f2bc711f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc00f5410 0x7f2bc010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:09.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.882+0000 7f2bc711f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc0107240 con 0x7f2bc00f5410 2023-12-21T03:43:09.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.882+0000 7f2bc4ebb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc00f5410 0x7f2bc010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:09.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.882+0000 7f2bc4ebb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc00f5410 0x7f2bc010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:48110/0 (socket says 172.21.15.148:48110) 2023-12-21T03:43:09.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.882+0000 7f2bc4ebb700 1 -- 172.21.15.148:0/2824497018 learned_addr learned my addr 172.21.15.148:0/2824497018 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:09.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.883+0000 7f2bc4ebb700 1 -- 172.21.15.148:0/2824497018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb4005040 con 0x7f2bc00f5410 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.883+0000 7f2bc4ebb700 1 --2- 172.21.15.148:0/2824497018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc00f5410 0x7f2bc010b850 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2bb4014000 tx=0x7f2bb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.883+0000 7f2bbd7fa700 1 -- 172.21.15.148:0/2824497018 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb40095f0 con 0x7f2bc00f5410 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.883+0000 7f2bbd7fa700 1 -- 172.21.15.148:0/2824497018 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bb4007e70 con 0x7f2bc00f5410 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.883+0000 7f2bc711f700 1 -- 172.21.15.148:0/2824497018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc0060220 con 0x7f2bc00f5410 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.883+0000 7f2bbd7fa700 1 -- 172.21.15.148:0/2824497018 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb4014ea0 con 0x7f2bc00f5410 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.883+0000 7f2bc711f700 1 -- 172.21.15.148:0/2824497018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc010bd40 con 0x7f2bc00f5410 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.884+0000 7f2bbd7fa700 1 -- 172.21.15.148:0/2824497018 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2bb4025070 con 0x7f2bc00f5410 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.884+0000 7f2bc711f700 1 -- 172.21.15.148:0/2824497018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc0045ad0 con 0x7f2bc00f5410 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.884+0000 7f2bbd7fa700 1 --2- 172.21.15.148:0/2824497018 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2bb002ff60 0x7f2bb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.884+0000 7f2bbd7fa700 1 -- 172.21.15.148:0/2824497018 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2bb40409f0 con 0x7f2bc00f5410 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.884+0000 7f2bbffff700 1 --2- 172.21.15.148:0/2824497018 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2bb002ff60 0x7f2bb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.885+0000 7f2bbffff700 1 --2- 172.21.15.148:0/2824497018 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2bb002ff60 0x7f2bb0032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f2bac0096a0 tx=0x7f2bac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:09.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:09.889+0000 7f2bbd7fa700 1 -- 172.21.15.148:0/2824497018 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bb400f4e0 con 0x7f2bc00f5410 2023-12-21T03:43:10.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.094+0000 7f2bc711f700 1 -- 172.21.15.148:0/2824497018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bc00fae60 con 0x7f2bc00f5410 2023-12-21T03:43:10.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.094+0000 7f2bbd7fa700 1 -- 172.21.15.148:0/2824497018 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2bb400f300 con 0x7f2bc00f5410 2023-12-21T03:43:10.097 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:10.097 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:10.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.096+0000 7f2bc711f700 1 -- 172.21.15.148:0/2824497018 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2bb002ff60 msgr2=0x7f2bb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:10.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.096+0000 7f2bc711f700 1 --2- 172.21.15.148:0/2824497018 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2bb002ff60 0x7f2bb0032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f2bac0096a0 tx=0x7f2bac006b40).stop 2023-12-21T03:43:10.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.096+0000 7f2bc711f700 1 -- 172.21.15.148:0/2824497018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc00f5410 msgr2=0x7f2bc010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:10.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.096+0000 7f2bc711f700 1 --2- 172.21.15.148:0/2824497018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc00f5410 0x7f2bc010b850 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2bb4014000 tx=0x7f2bb400b040).stop 2023-12-21T03:43:10.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.096+0000 7f2bc711f700 1 -- 172.21.15.148:0/2824497018 shutdown_connections 2023-12-21T03:43:10.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.096+0000 7f2bc711f700 1 --2- 172.21.15.148:0/2824497018 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2bb002ff60 0x7f2bb0032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:10.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.096+0000 7f2bc711f700 1 --2- 172.21.15.148:0/2824497018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc00f5410 0x7f2bc010b850 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:10.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.096+0000 7f2bc711f700 1 -- 172.21.15.148:0/2824497018 >> 172.21.15.148:0/2824497018 conn(0x7f2bc00f0b70 msgr2=0x7f2bc00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:10.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.096+0000 7f2bc711f700 1 -- 172.21.15.148:0/2824497018 shutdown_connections 2023-12-21T03:43:10.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:10.096+0000 7f2bc711f700 1 -- 172.21.15.148:0/2824497018 wait complete. 2023-12-21T03:43:10.099 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:10.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:10 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:10.096171+0000 mon.smithi078 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.148:0/2824497018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:11.513 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:11.513 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:11.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:10.436129+0000 mgr.smithi078.zidbgq (mgr.14162) 117 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:12.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.673+0000 7fa14b59e700 1 -- 172.21.15.148:0/3412306857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa14c0f51a0 msgr2=0x7fa14c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:12.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.673+0000 7fa14b59e700 1 --2- 172.21.15.148:0/3412306857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa14c0f51a0 0x7fa14c0f55c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa13c009fb0 tx=0x7fa13c009b30).stop 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.673+0000 7fa14b59e700 1 -- 172.21.15.148:0/3412306857 shutdown_connections 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.673+0000 7fa14b59e700 1 --2- 172.21.15.148:0/3412306857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa14c0f51a0 0x7fa14c0f55c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.673+0000 7fa14b59e700 1 -- 172.21.15.148:0/3412306857 >> 172.21.15.148:0/3412306857 conn(0x7fa14c0f09a0 msgr2=0x7fa14c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.673+0000 7fa14b59e700 1 -- 172.21.15.148:0/3412306857 shutdown_connections 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.673+0000 7fa14b59e700 1 -- 172.21.15.148:0/3412306857 wait complete. 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.673+0000 7fa14b59e700 1 Processor -- start 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.673+0000 7fa14b59e700 1 -- start start 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.673+0000 7fa14b59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa14c0f51a0 0x7fa14c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.674+0000 7fa14b59e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c060240 con 0x7fa14c0f51a0 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.674+0000 7fa14a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa14c0f51a0 0x7fa14c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.674+0000 7fa14a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa14c0f51a0 0x7fa14c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:48126/0 (socket says 172.21.15.148:48126) 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.674+0000 7fa14a59c700 1 -- 172.21.15.148:0/3865787597 learned_addr learned my addr 172.21.15.148:0/3865787597 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:12.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.674+0000 7fa14a59c700 1 -- 172.21.15.148:0/3865787597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c005040 con 0x7fa14c0f51a0 2023-12-21T03:43:12.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.674+0000 7fa14a59c700 1 --2- 172.21.15.148:0/3865787597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa14c0f51a0 0x7fa14c1091b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fa13c009830 tx=0x7fa13c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:12.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.674+0000 7fa13affd700 1 -- 172.21.15.148:0/3865787597 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa13c009320 con 0x7fa14c0f51a0 2023-12-21T03:43:12.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.674+0000 7fa13affd700 1 -- 172.21.15.148:0/3865787597 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa13c0136c0 con 0x7fa14c0f51a0 2023-12-21T03:43:12.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.674+0000 7fa14b59e700 1 -- 172.21.15.148:0/3865787597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14c060440 con 0x7fa14c0f51a0 2023-12-21T03:43:12.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.674+0000 7fa13affd700 1 -- 172.21.15.148:0/3865787597 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa13c01ee90 con 0x7fa14c0f51a0 2023-12-21T03:43:12.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.674+0000 7fa14b59e700 1 -- 172.21.15.148:0/3865787597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14c109860 con 0x7fa14c0f51a0 2023-12-21T03:43:12.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.675+0000 7fa13affd700 1 -- 172.21.15.148:0/3865787597 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa13c01e420 con 0x7fa14c0f51a0 2023-12-21T03:43:12.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.675+0000 7fa14b59e700 1 -- 172.21.15.148:0/3865787597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14c045ad0 con 0x7fa14c0f51a0 2023-12-21T03:43:12.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.675+0000 7fa13affd700 1 --2- 172.21.15.148:0/3865787597 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa13402ff10 0x7fa1340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:12.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.675+0000 7fa13affd700 1 -- 172.21.15.148:0/3865787597 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa13c040f10 con 0x7fa14c0f51a0 2023-12-21T03:43:12.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.676+0000 7fa149d9b700 1 --2- 172.21.15.148:0/3865787597 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa13402ff10 0x7fa1340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:12.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.676+0000 7fa149d9b700 1 --2- 172.21.15.148:0/3865787597 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa13402ff10 0x7fa1340323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa1400096a0 tx=0x7fa140006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:12.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.681+0000 7fa13affd700 1 -- 172.21.15.148:0/3865787597 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa13c0095f0 con 0x7fa14c0f51a0 2023-12-21T03:43:12.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:12.436467+0000 mgr.smithi078.zidbgq (mgr.14162) 118 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:12.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.887+0000 7fa14b59e700 1 -- 172.21.15.148:0/3865787597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa14c10a080 con 0x7fa14c0f51a0 2023-12-21T03:43:12.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.888+0000 7fa13affd700 1 -- 172.21.15.148:0/3865787597 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa13c02c2d0 con 0x7fa14c0f51a0 2023-12-21T03:43:12.890 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:12.890 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:12.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.889+0000 7fa14b59e700 1 -- 172.21.15.148:0/3865787597 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa13402ff10 msgr2=0x7fa1340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:12.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.889+0000 7fa14b59e700 1 --2- 172.21.15.148:0/3865787597 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa13402ff10 0x7fa1340323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa1400096a0 tx=0x7fa140006b40).stop 2023-12-21T03:43:12.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.889+0000 7fa14b59e700 1 -- 172.21.15.148:0/3865787597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa14c0f51a0 msgr2=0x7fa14c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:12.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.889+0000 7fa14b59e700 1 --2- 172.21.15.148:0/3865787597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa14c0f51a0 0x7fa14c1091b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fa13c009830 tx=0x7fa13c00b040).stop 2023-12-21T03:43:12.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.889+0000 7fa14b59e700 1 -- 172.21.15.148:0/3865787597 shutdown_connections 2023-12-21T03:43:12.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.889+0000 7fa14b59e700 1 --2- 172.21.15.148:0/3865787597 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa13402ff10 0x7fa1340323d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:12.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.889+0000 7fa14b59e700 1 --2- 172.21.15.148:0/3865787597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa14c0f51a0 0x7fa14c1091b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:12.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.889+0000 7fa14b59e700 1 -- 172.21.15.148:0/3865787597 >> 172.21.15.148:0/3865787597 conn(0x7fa14c0f09a0 msgr2=0x7fa14c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:12.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.889+0000 7fa14b59e700 1 -- 172.21.15.148:0/3865787597 shutdown_connections 2023-12-21T03:43:12.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:12.889+0000 7fa14b59e700 1 -- 172.21.15.148:0/3865787597 wait complete. 2023-12-21T03:43:12.892 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:12.889747+0000 mon.smithi078 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.148:0/3865787597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:14.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:14.280 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.489+0000 7f23d3b72700 1 -- 172.21.15.148:0/594034153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23cc0f5410 msgr2=0x7f23cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.489+0000 7f23d3b72700 1 --2- 172.21.15.148:0/594034153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23cc0f5410 0x7f23cc0f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f23c8009fb0 tx=0x7f23c8009b30).stop 2023-12-21T03:43:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.489+0000 7f23d3b72700 1 -- 172.21.15.148:0/594034153 shutdown_connections 2023-12-21T03:43:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.489+0000 7f23d3b72700 1 --2- 172.21.15.148:0/594034153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23cc0f5410 0x7f23cc0f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.489+0000 7f23d3b72700 1 -- 172.21.15.148:0/594034153 >> 172.21.15.148:0/594034153 conn(0x7f23cc0f0b70 msgr2=0x7f23cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.489+0000 7f23d3b72700 1 -- 172.21.15.148:0/594034153 shutdown_connections 2023-12-21T03:43:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.489+0000 7f23d3b72700 1 -- 172.21.15.148:0/594034153 wait complete. 2023-12-21T03:43:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.489+0000 7f23d3b72700 1 Processor -- start 2023-12-21T03:43:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.490+0000 7f23d3b72700 1 -- start start 2023-12-21T03:43:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.490+0000 7f23d3b72700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23cc0f5410 0x7f23cc105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.490+0000 7f23d3b72700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc105af0 con 0x7f23cc0f5410 2023-12-21T03:43:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.490+0000 7f23d190e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23cc0f5410 0x7f23cc105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.490+0000 7f23d190e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23cc0f5410 0x7f23cc105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:48156/0 (socket says 172.21.15.148:48156) 2023-12-21T03:43:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.490+0000 7f23d190e700 1 -- 172.21.15.148:0/131443235 learned_addr learned my addr 172.21.15.148:0/131443235 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.490+0000 7f23d190e700 1 -- 172.21.15.148:0/131443235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c8005040 con 0x7f23cc0f5410 2023-12-21T03:43:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.490+0000 7f23d190e700 1 --2- 172.21.15.148:0/131443235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23cc0f5410 0x7f23cc105600 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f23c8013510 tx=0x7f23c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.490+0000 7f23c27fc700 1 -- 172.21.15.148:0/131443235 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23c8009320 con 0x7f23cc0f5410 2023-12-21T03:43:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.491+0000 7f23d3b72700 1 -- 172.21.15.148:0/131443235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc105cf0 con 0x7f23cc0f5410 2023-12-21T03:43:15.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.491+0000 7f23c27fc700 1 -- 172.21.15.148:0/131443235 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23c8013f00 con 0x7f23cc0f5410 2023-12-21T03:43:15.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.491+0000 7f23c27fc700 1 -- 172.21.15.148:0/131443235 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23c8014d90 con 0x7f23cc0f5410 2023-12-21T03:43:15.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.491+0000 7f23d3b72700 1 -- 172.21.15.148:0/131443235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc106180 con 0x7f23cc0f5410 2023-12-21T03:43:15.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.491+0000 7f23d3b72700 1 -- 172.21.15.148:0/131443235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc045ad0 con 0x7f23cc0f5410 2023-12-21T03:43:15.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.492+0000 7f23c27fc700 1 -- 172.21.15.148:0/131443235 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23c80095f0 con 0x7f23cc0f5410 2023-12-21T03:43:15.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.492+0000 7f23c27fc700 1 --2- 172.21.15.148:0/131443235 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f23b802ff60 0x7f23b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:15.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.492+0000 7f23c27fc700 1 -- 172.21.15.148:0/131443235 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23c8041000 con 0x7f23cc0f5410 2023-12-21T03:43:15.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.492+0000 7f23d110d700 1 --2- 172.21.15.148:0/131443235 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f23b802ff60 0x7f23b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:15.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.492+0000 7f23d110d700 1 --2- 172.21.15.148:0/131443235 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f23b802ff60 0x7f23b8032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f23bc0096a0 tx=0x7f23bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:15.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.496+0000 7f23c27fc700 1 -- 172.21.15.148:0/131443235 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23c8014420 con 0x7f23cc0f5410 2023-12-21T03:43:15.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:14.436680+0000 mgr.smithi078.zidbgq (mgr.14162) 119 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:15.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.700+0000 7f23d3b72700 1 -- 172.21.15.148:0/131443235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23cc0fae60 con 0x7f23cc0f5410 2023-12-21T03:43:15.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.701+0000 7f23c27fc700 1 -- 172.21.15.148:0/131443235 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f23c801d7f0 con 0x7f23cc0f5410 2023-12-21T03:43:15.702 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:15.702 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:15.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.702+0000 7f23d3b72700 1 -- 172.21.15.148:0/131443235 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f23b802ff60 msgr2=0x7f23b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:15.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.702+0000 7f23d3b72700 1 --2- 172.21.15.148:0/131443235 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f23b802ff60 0x7f23b8032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f23bc0096a0 tx=0x7f23bc006b40).stop 2023-12-21T03:43:15.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.702+0000 7f23d3b72700 1 -- 172.21.15.148:0/131443235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23cc0f5410 msgr2=0x7f23cc105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:15.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.702+0000 7f23d3b72700 1 --2- 172.21.15.148:0/131443235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23cc0f5410 0x7f23cc105600 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f23c8013510 tx=0x7f23c800b040).stop 2023-12-21T03:43:15.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.702+0000 7f23d3b72700 1 -- 172.21.15.148:0/131443235 shutdown_connections 2023-12-21T03:43:15.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.702+0000 7f23d3b72700 1 --2- 172.21.15.148:0/131443235 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f23b802ff60 0x7f23b8032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:15.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.702+0000 7f23d3b72700 1 --2- 172.21.15.148:0/131443235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23cc0f5410 0x7f23cc105600 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:15.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.702+0000 7f23d3b72700 1 -- 172.21.15.148:0/131443235 >> 172.21.15.148:0/131443235 conn(0x7f23cc0f0b70 msgr2=0x7f23cc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:15.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.702+0000 7f23d3b72700 1 -- 172.21.15.148:0/131443235 shutdown_connections 2023-12-21T03:43:15.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:15.702+0000 7f23d3b72700 1 -- 172.21.15.148:0/131443235 wait complete. 2023-12-21T03:43:15.704 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:16.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:16 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:15.701399+0000 mon.smithi078 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.148:0/131443235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:17.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:17.078 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:17.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:16.436963+0000 mgr.smithi078.zidbgq (mgr.14162) 120 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:18.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.170+0000 7fd40ef96700 1 -- 172.21.15.148:0/2877437674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4080f5410 msgr2=0x7fd4080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:18.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.170+0000 7fd40ef96700 1 --2- 172.21.15.148:0/2877437674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4080f5410 0x7fd4080f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fd3fc009fb0 tx=0x7fd3fc009b30).stop 2023-12-21T03:43:18.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.170+0000 7fd40ef96700 1 -- 172.21.15.148:0/2877437674 shutdown_connections 2023-12-21T03:43:18.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.170+0000 7fd40ef96700 1 --2- 172.21.15.148:0/2877437674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4080f5410 0x7fd4080f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:18.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.170+0000 7fd40ef96700 1 -- 172.21.15.148:0/2877437674 >> 172.21.15.148:0/2877437674 conn(0x7fd4080f0b70 msgr2=0x7fd4080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:18.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.170+0000 7fd40ef96700 1 -- 172.21.15.148:0/2877437674 shutdown_connections 2023-12-21T03:43:18.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.170+0000 7fd40ef96700 1 -- 172.21.15.148:0/2877437674 wait complete. 2023-12-21T03:43:18.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.171+0000 7fd40ef96700 1 Processor -- start 2023-12-21T03:43:18.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.171+0000 7fd40ef96700 1 -- start start 2023-12-21T03:43:18.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.171+0000 7fd40ef96700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4080f5410 0x7fd408105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:18.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.171+0000 7fd40ef96700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd408105af0 con 0x7fd4080f5410 2023-12-21T03:43:18.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.171+0000 7fd40cd32700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4080f5410 0x7fd408105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:18.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.171+0000 7fd40cd32700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4080f5410 0x7fd408105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:48180/0 (socket says 172.21.15.148:48180) 2023-12-21T03:43:18.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.171+0000 7fd40cd32700 1 -- 172.21.15.148:0/1437121469 learned_addr learned my addr 172.21.15.148:0/1437121469 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:18.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.172+0000 7fd40cd32700 1 -- 172.21.15.148:0/1437121469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc005040 con 0x7fd4080f5410 2023-12-21T03:43:18.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.172+0000 7fd40cd32700 1 --2- 172.21.15.148:0/1437121469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4080f5410 0x7fd408105600 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fd3fc013510 tx=0x7fd3fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:18.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.172+0000 7fd4057fa700 1 -- 172.21.15.148:0/1437121469 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3fc009320 con 0x7fd4080f5410 2023-12-21T03:43:18.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.172+0000 7fd4057fa700 1 -- 172.21.15.148:0/1437121469 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3fc013f00 con 0x7fd4080f5410 2023-12-21T03:43:18.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.172+0000 7fd40ef96700 1 -- 172.21.15.148:0/1437121469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd408105cf0 con 0x7fd4080f5410 2023-12-21T03:43:18.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.172+0000 7fd4057fa700 1 -- 172.21.15.148:0/1437121469 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3fc014d90 con 0x7fd4080f5410 2023-12-21T03:43:18.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.172+0000 7fd40ef96700 1 -- 172.21.15.148:0/1437121469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd408106180 con 0x7fd4080f5410 2023-12-21T03:43:18.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.173+0000 7fd4057fa700 1 -- 172.21.15.148:0/1437121469 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3fc0263a0 con 0x7fd4080f5410 2023-12-21T03:43:18.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.173+0000 7fd4057fa700 1 --2- 172.21.15.148:0/1437121469 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3f802ff60 0x7fd3f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:18.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.173+0000 7fd40ef96700 1 -- 172.21.15.148:0/1437121469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd408045ad0 con 0x7fd4080f5410 2023-12-21T03:43:18.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.173+0000 7fd4057fa700 1 -- 172.21.15.148:0/1437121469 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3fc041300 con 0x7fd4080f5410 2023-12-21T03:43:18.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.173+0000 7fd407fff700 1 --2- 172.21.15.148:0/1437121469 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3f802ff60 0x7fd3f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:18.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.174+0000 7fd407fff700 1 --2- 172.21.15.148:0/1437121469 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3f802ff60 0x7fd3f8032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fd3f40096a0 tx=0x7fd3f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:18.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.178+0000 7fd4057fa700 1 -- 172.21.15.148:0/1437121469 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3fc012330 con 0x7fd4080f5410 2023-12-21T03:43:18.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.386+0000 7fd40ef96700 1 -- 172.21.15.148:0/1437121469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4080faf50 con 0x7fd4080f5410 2023-12-21T03:43:18.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.387+0000 7fd4057fa700 1 -- 172.21.15.148:0/1437121469 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd3fc03a030 con 0x7fd4080f5410 2023-12-21T03:43:18.388 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:18.388 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:18.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.388+0000 7fd40ef96700 1 -- 172.21.15.148:0/1437121469 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3f802ff60 msgr2=0x7fd3f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:18.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.388+0000 7fd40ef96700 1 --2- 172.21.15.148:0/1437121469 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3f802ff60 0x7fd3f8032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fd3f40096a0 tx=0x7fd3f4006b40).stop 2023-12-21T03:43:18.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.388+0000 7fd40ef96700 1 -- 172.21.15.148:0/1437121469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4080f5410 msgr2=0x7fd408105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:18.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.388+0000 7fd40ef96700 1 --2- 172.21.15.148:0/1437121469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4080f5410 0x7fd408105600 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fd3fc013510 tx=0x7fd3fc00b040).stop 2023-12-21T03:43:18.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.388+0000 7fd40ef96700 1 -- 172.21.15.148:0/1437121469 shutdown_connections 2023-12-21T03:43:18.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.388+0000 7fd40ef96700 1 --2- 172.21.15.148:0/1437121469 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3f802ff60 0x7fd3f8032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:18.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.388+0000 7fd40ef96700 1 --2- 172.21.15.148:0/1437121469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4080f5410 0x7fd408105600 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:18.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.388+0000 7fd40ef96700 1 -- 172.21.15.148:0/1437121469 >> 172.21.15.148:0/1437121469 conn(0x7fd4080f0b70 msgr2=0x7fd4080f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:18.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.388+0000 7fd40ef96700 1 -- 172.21.15.148:0/1437121469 shutdown_connections 2023-12-21T03:43:18.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:18.388+0000 7fd40ef96700 1 -- 172.21.15.148:0/1437121469 wait complete. 2023-12-21T03:43:18.390 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:18.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:18 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:18.387380+0000 mon.smithi078 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.148:0/1437121469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:19.753 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:19.754 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:19.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:18.437313+0000 mgr.smithi078.zidbgq (mgr.14162) 121 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:20.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.903+0000 7f5e92490700 1 -- 172.21.15.148:0/3505185680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e8c0f5410 msgr2=0x7f5e8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:20.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.903+0000 7f5e92490700 1 --2- 172.21.15.148:0/3505185680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e8c0f5410 0x7f5e8c0f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f5e7c009fb0 tx=0x7f5e7c009b30).stop 2023-12-21T03:43:20.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.903+0000 7f5e92490700 1 -- 172.21.15.148:0/3505185680 shutdown_connections 2023-12-21T03:43:20.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.903+0000 7f5e92490700 1 --2- 172.21.15.148:0/3505185680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e8c0f5410 0x7f5e8c0f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:20.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.903+0000 7f5e92490700 1 -- 172.21.15.148:0/3505185680 >> 172.21.15.148:0/3505185680 conn(0x7f5e8c0f0b70 msgr2=0x7f5e8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:20.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.903+0000 7f5e92490700 1 -- 172.21.15.148:0/3505185680 shutdown_connections 2023-12-21T03:43:20.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.903+0000 7f5e92490700 1 -- 172.21.15.148:0/3505185680 wait complete. 2023-12-21T03:43:20.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.904+0000 7f5e92490700 1 Processor -- start 2023-12-21T03:43:20.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.904+0000 7f5e92490700 1 -- start start 2023-12-21T03:43:20.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.904+0000 7f5e92490700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e8c0f5410 0x7f5e8c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:20.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.904+0000 7f5e92490700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c106c10 con 0x7f5e8c0f5410 2023-12-21T03:43:20.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.905+0000 7f5e8bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e8c0f5410 0x7f5e8c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:20.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.905+0000 7f5e8bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e8c0f5410 0x7f5e8c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:60984/0 (socket says 172.21.15.148:60984) 2023-12-21T03:43:20.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.905+0000 7f5e8bfff700 1 -- 172.21.15.148:0/770489759 learned_addr learned my addr 172.21.15.148:0/770489759 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:20.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.905+0000 7f5e8bfff700 1 -- 172.21.15.148:0/770489759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c005040 con 0x7f5e8c0f5410 2023-12-21T03:43:20.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.905+0000 7f5e8bfff700 1 --2- 172.21.15.148:0/770489759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e8c0f5410 0x7f5e8c106720 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f5e7c014000 tx=0x7f5e7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:20.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.905+0000 7f5e88ff9700 1 -- 172.21.15.148:0/770489759 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e7c0095f0 con 0x7f5e8c0f5410 2023-12-21T03:43:20.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.905+0000 7f5e92490700 1 -- 172.21.15.148:0/770489759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8c106e10 con 0x7f5e8c0f5410 2023-12-21T03:43:20.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.905+0000 7f5e88ff9700 1 -- 172.21.15.148:0/770489759 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e7c007e70 con 0x7f5e8c0f5410 2023-12-21T03:43:20.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.905+0000 7f5e88ff9700 1 -- 172.21.15.148:0/770489759 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e7c014ea0 con 0x7f5e8c0f5410 2023-12-21T03:43:20.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.905+0000 7f5e92490700 1 -- 172.21.15.148:0/770489759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8c1072a0 con 0x7f5e8c0f5410 2023-12-21T03:43:20.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.906+0000 7f5e88ff9700 1 -- 172.21.15.148:0/770489759 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e7c025070 con 0x7f5e8c0f5410 2023-12-21T03:43:20.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.906+0000 7f5e92490700 1 -- 172.21.15.148:0/770489759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8c045ad0 con 0x7f5e8c0f5410 2023-12-21T03:43:20.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.906+0000 7f5e88ff9700 1 --2- 172.21.15.148:0/770489759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e7402ff60 0x7f5e74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:20.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.906+0000 7f5e88ff9700 1 -- 172.21.15.148:0/770489759 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e7c040940 con 0x7f5e8c0f5410 2023-12-21T03:43:20.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.907+0000 7f5e8b7fe700 1 --2- 172.21.15.148:0/770489759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e7402ff60 0x7f5e74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:20.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.907+0000 7f5e8b7fe700 1 --2- 172.21.15.148:0/770489759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e7402ff60 0x7f5e74032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f5e800096a0 tx=0x7f5e80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:20.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:20.911+0000 7f5e88ff9700 1 -- 172.21.15.148:0/770489759 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e7c0079c0 con 0x7f5e8c0f5410 2023-12-21T03:43:21.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.117+0000 7f5e92490700 1 -- 172.21.15.148:0/770489759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e8c1080d0 con 0x7f5e8c0f5410 2023-12-21T03:43:21.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.117+0000 7f5e88ff9700 1 -- 172.21.15.148:0/770489759 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e7c00f300 con 0x7f5e8c0f5410 2023-12-21T03:43:21.118 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:21.119 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:21.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.119+0000 7f5e92490700 1 -- 172.21.15.148:0/770489759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e7402ff60 msgr2=0x7f5e74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:21.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.119+0000 7f5e92490700 1 --2- 172.21.15.148:0/770489759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e7402ff60 0x7f5e74032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f5e800096a0 tx=0x7f5e80006b40).stop 2023-12-21T03:43:21.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.119+0000 7f5e92490700 1 -- 172.21.15.148:0/770489759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e8c0f5410 msgr2=0x7f5e8c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:21.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.119+0000 7f5e92490700 1 --2- 172.21.15.148:0/770489759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e8c0f5410 0x7f5e8c106720 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f5e7c014000 tx=0x7f5e7c00b040).stop 2023-12-21T03:43:21.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.119+0000 7f5e92490700 1 -- 172.21.15.148:0/770489759 shutdown_connections 2023-12-21T03:43:21.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.119+0000 7f5e92490700 1 --2- 172.21.15.148:0/770489759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5e7402ff60 0x7f5e74032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:21.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.119+0000 7f5e92490700 1 --2- 172.21.15.148:0/770489759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e8c0f5410 0x7f5e8c106720 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:21.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.119+0000 7f5e92490700 1 -- 172.21.15.148:0/770489759 >> 172.21.15.148:0/770489759 conn(0x7f5e8c0f0b70 msgr2=0x7f5e8c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:21.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.119+0000 7f5e92490700 1 -- 172.21.15.148:0/770489759 shutdown_connections 2023-12-21T03:43:21.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:21.119+0000 7f5e92490700 1 -- 172.21.15.148:0/770489759 wait complete. 2023-12-21T03:43:21.121 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:20.437623+0000 mgr.smithi078.zidbgq (mgr.14162) 122 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:21.118150+0000 mon.smithi078 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.148:0/770489759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:22.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:22.519 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:23.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.620+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/307758492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d780f51a0 msgr2=0x7f8d780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:23.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.620+0000 7f8d7dfbe700 1 --2- 172.21.15.148:0/307758492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d780f51a0 0x7f8d780f55c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f8d6c009fb0 tx=0x7f8d6c009b30).stop 2023-12-21T03:43:23.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.620+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/307758492 shutdown_connections 2023-12-21T03:43:23.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.620+0000 7f8d7dfbe700 1 --2- 172.21.15.148:0/307758492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d780f51a0 0x7f8d780f55c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:23.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.620+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/307758492 >> 172.21.15.148:0/307758492 conn(0x7f8d780f09a0 msgr2=0x7f8d780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:23.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.620+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/307758492 shutdown_connections 2023-12-21T03:43:23.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.620+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/307758492 wait complete. 2023-12-21T03:43:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.620+0000 7f8d7dfbe700 1 Processor -- start 2023-12-21T03:43:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.620+0000 7f8d7dfbe700 1 -- start start 2023-12-21T03:43:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.621+0000 7f8d7dfbe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d780f51a0 0x7f8d781064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.621+0000 7f8d7dfbe700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d781069e0 con 0x7f8d780f51a0 2023-12-21T03:43:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.621+0000 7f8d7cfbc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d780f51a0 0x7f8d781064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.621+0000 7f8d7cfbc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d780f51a0 0x7f8d781064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:60990/0 (socket says 172.21.15.148:60990) 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.621+0000 7f8d7cfbc700 1 -- 172.21.15.148:0/3662468200 learned_addr learned my addr 172.21.15.148:0/3662468200 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.621+0000 7f8d7cfbc700 1 -- 172.21.15.148:0/3662468200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c005040 con 0x7f8d780f51a0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.621+0000 7f8d7cfbc700 1 --2- 172.21.15.148:0/3662468200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d780f51a0 0x7f8d781064f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f8d6c013510 tx=0x7f8d6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.621+0000 7f8d757fa700 1 -- 172.21.15.148:0/3662468200 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d6c009320 con 0x7f8d780f51a0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.621+0000 7f8d757fa700 1 -- 172.21.15.148:0/3662468200 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d6c013f00 con 0x7f8d780f51a0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.622+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/3662468200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d78106be0 con 0x7f8d780f51a0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.622+0000 7f8d757fa700 1 -- 172.21.15.148:0/3662468200 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d6c014dc0 con 0x7f8d780f51a0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.622+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/3662468200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d78107070 con 0x7f8d780f51a0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.622+0000 7f8d757fa700 1 -- 172.21.15.148:0/3662468200 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d6c0095f0 con 0x7f8d780f51a0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.622+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/3662468200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d78045ad0 con 0x7f8d780f51a0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.622+0000 7f8d757fa700 1 --2- 172.21.15.148:0/3662468200 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d6802ffb0 0x7f8d68032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.622+0000 7f8d757fa700 1 -- 172.21.15.148:0/3662468200 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d6c041070 con 0x7f8d780f51a0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.623+0000 7f8d77fff700 1 --2- 172.21.15.148:0/3662468200 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d6802ffb0 0x7f8d68032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.623+0000 7f8d77fff700 1 --2- 172.21.15.148:0/3662468200 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d6802ffb0 0x7f8d68032470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8d640096a0 tx=0x7f8d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:23.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.627+0000 7f8d757fa700 1 -- 172.21.15.148:0/3662468200 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d6c06d2b0 con 0x7f8d780f51a0 2023-12-21T03:43:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:22.437972+0000 mgr.smithi078.zidbgq (mgr.14162) 123 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:23.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.838+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/3662468200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d78107f20 con 0x7f8d780f51a0 2023-12-21T03:43:23.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.839+0000 7f8d757fa700 1 -- 172.21.15.148:0/3662468200 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d6c01d7f0 con 0x7f8d780f51a0 2023-12-21T03:43:23.840 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:23.840 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:23.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.841+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/3662468200 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d6802ffb0 msgr2=0x7f8d68032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:23.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.841+0000 7f8d7dfbe700 1 --2- 172.21.15.148:0/3662468200 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d6802ffb0 0x7f8d68032470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8d640096a0 tx=0x7f8d64006b40).stop 2023-12-21T03:43:23.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.841+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/3662468200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d780f51a0 msgr2=0x7f8d781064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:23.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.841+0000 7f8d7dfbe700 1 --2- 172.21.15.148:0/3662468200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d780f51a0 0x7f8d781064f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f8d6c013510 tx=0x7f8d6c00b040).stop 2023-12-21T03:43:23.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.841+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/3662468200 shutdown_connections 2023-12-21T03:43:23.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.841+0000 7f8d7dfbe700 1 --2- 172.21.15.148:0/3662468200 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d6802ffb0 0x7f8d68032470 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:23.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.841+0000 7f8d7dfbe700 1 --2- 172.21.15.148:0/3662468200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d780f51a0 0x7f8d781064f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:23.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.841+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/3662468200 >> 172.21.15.148:0/3662468200 conn(0x7f8d780f09a0 msgr2=0x7f8d780f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:23.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.841+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/3662468200 shutdown_connections 2023-12-21T03:43:23.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:23.841+0000 7f8d7dfbe700 1 -- 172.21.15.148:0/3662468200 wait complete. 2023-12-21T03:43:23.844 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:24.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:24 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:23.839990+0000 mon.smithi078 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.148:0/3662468200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:25.212 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:25.212 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:24.438350+0000 mgr.smithi078.zidbgq (mgr.14162) 124 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:26.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.385+0000 7ff99394e700 1 -- 172.21.15.148:0/46491326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff98c0f5410 msgr2=0x7ff98c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:26.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.385+0000 7ff99394e700 1 --2- 172.21.15.148:0/46491326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff98c0f5410 0x7ff98c0f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7ff97c009fb0 tx=0x7ff97c009b30).stop 2023-12-21T03:43:26.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.385+0000 7ff99394e700 1 -- 172.21.15.148:0/46491326 shutdown_connections 2023-12-21T03:43:26.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.385+0000 7ff99394e700 1 --2- 172.21.15.148:0/46491326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff98c0f5410 0x7ff98c0f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:26.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.385+0000 7ff99394e700 1 -- 172.21.15.148:0/46491326 >> 172.21.15.148:0/46491326 conn(0x7ff98c0f0b70 msgr2=0x7ff98c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:26.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.385+0000 7ff99394e700 1 -- 172.21.15.148:0/46491326 shutdown_connections 2023-12-21T03:43:26.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.385+0000 7ff99394e700 1 -- 172.21.15.148:0/46491326 wait complete. 2023-12-21T03:43:26.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.386+0000 7ff99394e700 1 Processor -- start 2023-12-21T03:43:26.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.386+0000 7ff99394e700 1 -- start start 2023-12-21T03:43:26.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.386+0000 7ff99394e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff98c0fd3e0 0x7ff98c0fb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.386+0000 7ff99394e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c0fd800 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.386+0000 7ff9916ea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff98c0fd3e0 0x7ff98c0fb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.386+0000 7ff9916ea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff98c0fd3e0 0x7ff98c0fb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:32782/0 (socket says 172.21.15.148:32782) 2023-12-21T03:43:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.386+0000 7ff9916ea700 1 -- 172.21.15.148:0/1253008855 learned_addr learned my addr 172.21.15.148:0/1253008855 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.387+0000 7ff9916ea700 1 -- 172.21.15.148:0/1253008855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c005040 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.387+0000 7ff9916ea700 1 --2- 172.21.15.148:0/1253008855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff98c0fd3e0 0x7ff98c0fb8a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7ff97c009f80 tx=0x7ff97c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.387+0000 7ff981ffb700 1 -- 172.21.15.148:0/1253008855 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff97c0136c0 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.387+0000 7ff99394e700 1 -- 172.21.15.148:0/1253008855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98c0fbd90 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.387+0000 7ff981ffb700 1 -- 172.21.15.148:0/1253008855 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff97c0095f0 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.387+0000 7ff981ffb700 1 -- 172.21.15.148:0/1253008855 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff97c014420 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.387+0000 7ff99394e700 1 -- 172.21.15.148:0/1253008855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98c0fc280 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.388+0000 7ff981ffb700 1 -- 172.21.15.148:0/1253008855 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff97c025060 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.388+0000 7ff99394e700 1 -- 172.21.15.148:0/1253008855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98c059100 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.388+0000 7ff981ffb700 1 --2- 172.21.15.148:0/1253008855 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff97802ffb0 0x7ff978032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:26.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.388+0000 7ff981ffb700 1 -- 172.21.15.148:0/1253008855 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff97c040d50 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.388+0000 7ff990ee9700 1 --2- 172.21.15.148:0/1253008855 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff97802ffb0 0x7ff978032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:26.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.389+0000 7ff990ee9700 1 --2- 172.21.15.148:0/1253008855 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff97802ffb0 0x7ff978032470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff9880096a0 tx=0x7ff988006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:26.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.393+0000 7ff981ffb700 1 -- 172.21.15.148:0/1253008855 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff97c01e960 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.601+0000 7ff99394e700 1 -- 172.21.15.148:0/1253008855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff98c0fc670 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.602+0000 7ff981ffb700 1 -- 172.21.15.148:0/1253008855 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff97c00f280 con 0x7ff98c0fd3e0 2023-12-21T03:43:26.603 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:26.603 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:26.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.604+0000 7ff99394e700 1 -- 172.21.15.148:0/1253008855 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff97802ffb0 msgr2=0x7ff978032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:26.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.604+0000 7ff99394e700 1 --2- 172.21.15.148:0/1253008855 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff97802ffb0 0x7ff978032470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff9880096a0 tx=0x7ff988006b40).stop 2023-12-21T03:43:26.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.604+0000 7ff99394e700 1 -- 172.21.15.148:0/1253008855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff98c0fd3e0 msgr2=0x7ff98c0fb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:26.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.604+0000 7ff99394e700 1 --2- 172.21.15.148:0/1253008855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff98c0fd3e0 0x7ff98c0fb8a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7ff97c009f80 tx=0x7ff97c009b30).stop 2023-12-21T03:43:26.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.604+0000 7ff99394e700 1 -- 172.21.15.148:0/1253008855 shutdown_connections 2023-12-21T03:43:26.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.604+0000 7ff99394e700 1 --2- 172.21.15.148:0/1253008855 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff97802ffb0 0x7ff978032470 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:26.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.604+0000 7ff99394e700 1 --2- 172.21.15.148:0/1253008855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff98c0fd3e0 0x7ff98c0fb8a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:26.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.604+0000 7ff99394e700 1 -- 172.21.15.148:0/1253008855 >> 172.21.15.148:0/1253008855 conn(0x7ff98c0f0b70 msgr2=0x7ff98c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:26.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.604+0000 7ff99394e700 1 -- 172.21.15.148:0/1253008855 shutdown_connections 2023-12-21T03:43:26.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:26.604+0000 7ff99394e700 1 -- 172.21.15.148:0/1253008855 wait complete. 2023-12-21T03:43:26.606 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:26.438701+0000 mgr.smithi078.zidbgq (mgr.14162) 125 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:26.602546+0000 mon.smithi078 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.148:0/1253008855' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:27.996 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:27.997 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:29.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.088+0000 7f4769398700 1 -- 172.21.15.148:0/2367122023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47640f6510 msgr2=0x7f47640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:29.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.088+0000 7f4769398700 1 --2- 172.21.15.148:0/2367122023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47640f6510 0x7f47640f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f474c009fb0 tx=0x7f474c009b30).stop 2023-12-21T03:43:29.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.088+0000 7f4769398700 1 -- 172.21.15.148:0/2367122023 shutdown_connections 2023-12-21T03:43:29.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.088+0000 7f4769398700 1 --2- 172.21.15.148:0/2367122023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47640f6510 0x7f47640f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:29.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.088+0000 7f4769398700 1 -- 172.21.15.148:0/2367122023 >> 172.21.15.148:0/2367122023 conn(0x7f47640f1c70 msgr2=0x7f47640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:29.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.088+0000 7f4769398700 1 -- 172.21.15.148:0/2367122023 shutdown_connections 2023-12-21T03:43:29.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.088+0000 7f4769398700 1 -- 172.21.15.148:0/2367122023 wait complete. 2023-12-21T03:43:29.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.089+0000 7f4769398700 1 Processor -- start 2023-12-21T03:43:29.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.089+0000 7f4769398700 1 -- start start 2023-12-21T03:43:29.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.089+0000 7f4769398700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47640f6510 0x7f4764106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:29.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.089+0000 7f4769398700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4764106c20 con 0x7f47640f6510 2023-12-21T03:43:29.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.090+0000 7f4762ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47640f6510 0x7f4764106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:29.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.090+0000 7f4762ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47640f6510 0x7f4764106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:32810/0 (socket says 172.21.15.148:32810) 2023-12-21T03:43:29.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.090+0000 7f4762ffd700 1 -- 172.21.15.148:0/1185873352 learned_addr learned my addr 172.21.15.148:0/1185873352 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:29.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.090+0000 7f4762ffd700 1 -- 172.21.15.148:0/1185873352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c005040 con 0x7f47640f6510 2023-12-21T03:43:29.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.090+0000 7f4762ffd700 1 --2- 172.21.15.148:0/1185873352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47640f6510 0x7f4764106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f474c013510 tx=0x7f474c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:29.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.090+0000 7f475b7fe700 1 -- 172.21.15.148:0/1185873352 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474c009320 con 0x7f47640f6510 2023-12-21T03:43:29.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.090+0000 7f4769398700 1 -- 172.21.15.148:0/1185873352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4764106e20 con 0x7f47640f6510 2023-12-21T03:43:29.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.091+0000 7f475b7fe700 1 -- 172.21.15.148:0/1185873352 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f474c013f00 con 0x7f47640f6510 2023-12-21T03:43:29.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.091+0000 7f475b7fe700 1 -- 172.21.15.148:0/1185873352 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474c014d90 con 0x7f47640f6510 2023-12-21T03:43:29.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.091+0000 7f4769398700 1 -- 172.21.15.148:0/1185873352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47641072b0 con 0x7f47640f6510 2023-12-21T03:43:29.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.091+0000 7f4769398700 1 -- 172.21.15.148:0/1185873352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4764045ad0 con 0x7f47640f6510 2023-12-21T03:43:29.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.091+0000 7f475b7fe700 1 -- 172.21.15.148:0/1185873352 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f474c0095f0 con 0x7f47640f6510 2023-12-21T03:43:29.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.091+0000 7f475b7fe700 1 --2- 172.21.15.148:0/1185873352 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f475002ff60 0x7f4750032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:29.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.092+0000 7f475b7fe700 1 -- 172.21.15.148:0/1185873352 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f474c040fc0 con 0x7f47640f6510 2023-12-21T03:43:29.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.092+0000 7f47627fc700 1 --2- 172.21.15.148:0/1185873352 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f475002ff60 0x7f4750032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:29.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.092+0000 7f47627fc700 1 --2- 172.21.15.148:0/1185873352 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f475002ff60 0x7f4750032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f47540096a0 tx=0x7f4754006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:29.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.096+0000 7f475b7fe700 1 -- 172.21.15.148:0/1185873352 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f474c014420 con 0x7f47640f6510 2023-12-21T03:43:29.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.302+0000 7f4769398700 1 -- 172.21.15.148:0/1185873352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47640fbf60 con 0x7f47640f6510 2023-12-21T03:43:29.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.302+0000 7f475b7fe700 1 -- 172.21.15.148:0/1185873352 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f474c010230 con 0x7f47640f6510 2023-12-21T03:43:29.304 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:29.304 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:29.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.304+0000 7f4769398700 1 -- 172.21.15.148:0/1185873352 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f475002ff60 msgr2=0x7f4750032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:29.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.304+0000 7f4769398700 1 --2- 172.21.15.148:0/1185873352 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f475002ff60 0x7f4750032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f47540096a0 tx=0x7f4754006b40).stop 2023-12-21T03:43:29.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.304+0000 7f4769398700 1 -- 172.21.15.148:0/1185873352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47640f6510 msgr2=0x7f4764106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:29.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.304+0000 7f4769398700 1 --2- 172.21.15.148:0/1185873352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47640f6510 0x7f4764106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f474c013510 tx=0x7f474c00b040).stop 2023-12-21T03:43:29.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.304+0000 7f4769398700 1 -- 172.21.15.148:0/1185873352 shutdown_connections 2023-12-21T03:43:29.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.304+0000 7f4769398700 1 --2- 172.21.15.148:0/1185873352 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f475002ff60 0x7f4750032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:29.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.304+0000 7f4769398700 1 --2- 172.21.15.148:0/1185873352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47640f6510 0x7f4764106730 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:29.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.304+0000 7f4769398700 1 -- 172.21.15.148:0/1185873352 >> 172.21.15.148:0/1185873352 conn(0x7f47640f1c70 msgr2=0x7f47640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:29.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.304+0000 7f4769398700 1 -- 172.21.15.148:0/1185873352 shutdown_connections 2023-12-21T03:43:29.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:29.304+0000 7f4769398700 1 -- 172.21.15.148:0/1185873352 wait complete. 2023-12-21T03:43:29.307 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:29.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:28.438979+0000 mgr.smithi078.zidbgq (mgr.14162) 126 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:29.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:29.303062+0000 mon.smithi078 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.148:0/1185873352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:30.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:30.693 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:31.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.776+0000 7fd708aca700 1 -- 172.21.15.148:0/3977769397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7040f5410 msgr2=0x7fd7040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:31.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.776+0000 7fd708aca700 1 --2- 172.21.15.148:0/3977769397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7040f5410 0x7fd7040f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fd6f4009fb0 tx=0x7fd6f4009b30).stop 2023-12-21T03:43:31.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.776+0000 7fd708aca700 1 -- 172.21.15.148:0/3977769397 shutdown_connections 2023-12-21T03:43:31.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.776+0000 7fd708aca700 1 --2- 172.21.15.148:0/3977769397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7040f5410 0x7fd7040f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:31.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.776+0000 7fd708aca700 1 -- 172.21.15.148:0/3977769397 >> 172.21.15.148:0/3977769397 conn(0x7fd7040f0b70 msgr2=0x7fd7040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:31.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.776+0000 7fd708aca700 1 -- 172.21.15.148:0/3977769397 shutdown_connections 2023-12-21T03:43:31.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.776+0000 7fd708aca700 1 -- 172.21.15.148:0/3977769397 wait complete. 2023-12-21T03:43:31.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.776+0000 7fd708aca700 1 Processor -- start 2023-12-21T03:43:31.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.776+0000 7fd708aca700 1 -- start start 2023-12-21T03:43:31.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.777+0000 7fd708aca700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7040f5410 0x7fd704108930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:31.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.777+0000 7fd708aca700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd704108e20 con 0x7fd7040f5410 2023-12-21T03:43:31.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.777+0000 7fd70259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7040f5410 0x7fd704108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:31.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.777+0000 7fd70259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7040f5410 0x7fd704108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:55532/0 (socket says 172.21.15.148:55532) 2023-12-21T03:43:31.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.777+0000 7fd70259c700 1 -- 172.21.15.148:0/1177874997 learned_addr learned my addr 172.21.15.148:0/1177874997 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:31.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.777+0000 7fd70259c700 1 -- 172.21.15.148:0/1177874997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f4005040 con 0x7fd7040f5410 2023-12-21T03:43:31.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.777+0000 7fd70259c700 1 --2- 172.21.15.148:0/1177874997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7040f5410 0x7fd704108930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fd6f4013510 tx=0x7fd6f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:31.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.777+0000 7fd6f2ffd700 1 -- 172.21.15.148:0/1177874997 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6f4009320 con 0x7fd7040f5410 2023-12-21T03:43:31.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.777+0000 7fd708aca700 1 -- 172.21.15.148:0/1177874997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd704109020 con 0x7fd7040f5410 2023-12-21T03:43:31.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.777+0000 7fd6f2ffd700 1 -- 172.21.15.148:0/1177874997 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6f4013f00 con 0x7fd7040f5410 2023-12-21T03:43:31.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.777+0000 7fd6f2ffd700 1 -- 172.21.15.148:0/1177874997 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6f4014d90 con 0x7fd7040f5410 2023-12-21T03:43:31.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.778+0000 7fd708aca700 1 -- 172.21.15.148:0/1177874997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7041094b0 con 0x7fd7040f5410 2023-12-21T03:43:31.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.778+0000 7fd6f2ffd700 1 -- 172.21.15.148:0/1177874997 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6f40095f0 con 0x7fd7040f5410 2023-12-21T03:43:31.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.778+0000 7fd708aca700 1 -- 172.21.15.148:0/1177874997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd704045ad0 con 0x7fd7040f5410 2023-12-21T03:43:31.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.778+0000 7fd6f2ffd700 1 --2- 172.21.15.148:0/1177874997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd6ec02ffa0 0x7fd6ec032460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:31.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.778+0000 7fd6f2ffd700 1 -- 172.21.15.148:0/1177874997 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd6f4040fc0 con 0x7fd7040f5410 2023-12-21T03:43:31.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.779+0000 7fd701d9b700 1 --2- 172.21.15.148:0/1177874997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd6ec02ffa0 0x7fd6ec032460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:31.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.779+0000 7fd701d9b700 1 --2- 172.21.15.148:0/1177874997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd6ec02ffa0 0x7fd6ec032460 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fd6f80096a0 tx=0x7fd6f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:31.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.783+0000 7fd6f2ffd700 1 -- 172.21.15.148:0/1177874997 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd6f4014420 con 0x7fd7040f5410 2023-12-21T03:43:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:30.439283+0000 mgr.smithi078.zidbgq (mgr.14162) 127 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:30.543703+0000 mon.smithi078 (mon.0) 208 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:43:31.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:30.551254+0000 mon.smithi078 (mon.0) 209 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:43:31.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.988+0000 7fd708aca700 1 -- 172.21.15.148:0/1177874997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7040f9d80 con 0x7fd7040f5410 2023-12-21T03:43:31.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.989+0000 7fd6f2ffd700 1 -- 172.21.15.148:0/1177874997 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd6f401d7f0 con 0x7fd7040f5410 2023-12-21T03:43:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:31.990 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:31.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.991+0000 7fd708aca700 1 -- 172.21.15.148:0/1177874997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd6ec02ffa0 msgr2=0x7fd6ec032460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:31.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.991+0000 7fd708aca700 1 --2- 172.21.15.148:0/1177874997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd6ec02ffa0 0x7fd6ec032460 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fd6f80096a0 tx=0x7fd6f8006b40).stop 2023-12-21T03:43:31.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.991+0000 7fd708aca700 1 -- 172.21.15.148:0/1177874997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7040f5410 msgr2=0x7fd704108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:31.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.991+0000 7fd708aca700 1 --2- 172.21.15.148:0/1177874997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7040f5410 0x7fd704108930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fd6f4013510 tx=0x7fd6f400b040).stop 2023-12-21T03:43:31.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.991+0000 7fd708aca700 1 -- 172.21.15.148:0/1177874997 shutdown_connections 2023-12-21T03:43:31.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.991+0000 7fd708aca700 1 --2- 172.21.15.148:0/1177874997 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd6ec02ffa0 0x7fd6ec032460 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:31.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.991+0000 7fd708aca700 1 --2- 172.21.15.148:0/1177874997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7040f5410 0x7fd704108930 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:31.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.991+0000 7fd708aca700 1 -- 172.21.15.148:0/1177874997 >> 172.21.15.148:0/1177874997 conn(0x7fd7040f0b70 msgr2=0x7fd7040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:31.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.991+0000 7fd708aca700 1 -- 172.21.15.148:0/1177874997 shutdown_connections 2023-12-21T03:43:31.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:31.991+0000 7fd708aca700 1 -- 172.21.15.148:0/1177874997 wait complete. 2023-12-21T03:43:31.993 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:32.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:31.989715+0000 mon.smithi078 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.148:0/1177874997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:33.332 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:33.332 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:33.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:32.439510+0000 mgr.smithi078.zidbgq (mgr.14162) 128 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:34.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.495+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/89414388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f740f5410 msgr2=0x7f0f740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:34.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.495+0000 7f0f7b6ce700 1 --2- 172.21.15.148:0/89414388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f740f5410 0x7f0f740f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f0f68009fb0 tx=0x7f0f68009b30).stop 2023-12-21T03:43:34.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.495+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/89414388 shutdown_connections 2023-12-21T03:43:34.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.495+0000 7f0f7b6ce700 1 --2- 172.21.15.148:0/89414388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f740f5410 0x7f0f740f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:34.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.495+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/89414388 >> 172.21.15.148:0/89414388 conn(0x7f0f740f0b70 msgr2=0x7f0f740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:34.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.495+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/89414388 shutdown_connections 2023-12-21T03:43:34.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.495+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/89414388 wait complete. 2023-12-21T03:43:34.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.496+0000 7f0f7b6ce700 1 Processor -- start 2023-12-21T03:43:34.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.496+0000 7f0f7b6ce700 1 -- start start 2023-12-21T03:43:34.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.496+0000 7f0f7b6ce700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f740f5410 0x7f0f74109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:34.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.496+0000 7f0f7b6ce700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f74060240 con 0x7f0f740f5410 2023-12-21T03:43:34.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.496+0000 7f0f7946a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f740f5410 0x7f0f74109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:34.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.496+0000 7f0f7946a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f740f5410 0x7f0f74109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:55550/0 (socket says 172.21.15.148:55550) 2023-12-21T03:43:34.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.496+0000 7f0f7946a700 1 -- 172.21.15.148:0/1354509262 learned_addr learned my addr 172.21.15.148:0/1354509262 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:34.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.496+0000 7f0f7946a700 1 -- 172.21.15.148:0/1354509262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f68005040 con 0x7f0f740f5410 2023-12-21T03:43:34.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.496+0000 7f0f7946a700 1 --2- 172.21.15.148:0/1354509262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f740f5410 0x7f0f74109430 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0f68013510 tx=0x7f0f6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:34.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.497+0000 7f0f65ffb700 1 -- 172.21.15.148:0/1354509262 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f68009320 con 0x7f0f740f5410 2023-12-21T03:43:34.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.497+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/1354509262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f74060440 con 0x7f0f740f5410 2023-12-21T03:43:34.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.497+0000 7f0f65ffb700 1 -- 172.21.15.148:0/1354509262 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f68013f00 con 0x7f0f740f5410 2023-12-21T03:43:34.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.497+0000 7f0f65ffb700 1 -- 172.21.15.148:0/1354509262 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f68014d90 con 0x7f0f740f5410 2023-12-21T03:43:34.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.497+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/1354509262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f74109ae0 con 0x7f0f740f5410 2023-12-21T03:43:34.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.498+0000 7f0f65ffb700 1 -- 172.21.15.148:0/1354509262 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f68014420 con 0x7f0f740f5410 2023-12-21T03:43:34.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.498+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/1354509262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f74045ad0 con 0x7f0f740f5410 2023-12-21T03:43:34.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.498+0000 7f0f65ffb700 1 --2- 172.21.15.148:0/1354509262 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0f6002ff60 0x7f0f60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:34.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.498+0000 7f0f65ffb700 1 -- 172.21.15.148:0/1354509262 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f68041000 con 0x7f0f740f5410 2023-12-21T03:43:34.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.498+0000 7f0f78c69700 1 --2- 172.21.15.148:0/1354509262 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0f6002ff60 0x7f0f60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:34.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.499+0000 7f0f78c69700 1 --2- 172.21.15.148:0/1354509262 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0f6002ff60 0x7f0f60032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f0f700096a0 tx=0x7f0f70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:34.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.503+0000 7f0f65ffb700 1 -- 172.21.15.148:0/1354509262 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f680095f0 con 0x7f0f740f5410 2023-12-21T03:43:34.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.709+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/1354509262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f740fae60 con 0x7f0f740f5410 2023-12-21T03:43:34.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.710+0000 7f0f65ffb700 1 -- 172.21.15.148:0/1354509262 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f6801db70 con 0x7f0f740f5410 2023-12-21T03:43:34.711 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:34.711 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:34.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.712+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/1354509262 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0f6002ff60 msgr2=0x7f0f60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:34.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.712+0000 7f0f7b6ce700 1 --2- 172.21.15.148:0/1354509262 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0f6002ff60 0x7f0f60032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f0f700096a0 tx=0x7f0f70006b40).stop 2023-12-21T03:43:34.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.712+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/1354509262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f740f5410 msgr2=0x7f0f74109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:34.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.712+0000 7f0f7b6ce700 1 --2- 172.21.15.148:0/1354509262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f740f5410 0x7f0f74109430 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0f68013510 tx=0x7f0f6800b040).stop 2023-12-21T03:43:34.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.712+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/1354509262 shutdown_connections 2023-12-21T03:43:34.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.712+0000 7f0f7b6ce700 1 --2- 172.21.15.148:0/1354509262 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0f6002ff60 0x7f0f60032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:34.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.712+0000 7f0f7b6ce700 1 --2- 172.21.15.148:0/1354509262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f740f5410 0x7f0f74109430 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:34.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.712+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/1354509262 >> 172.21.15.148:0/1354509262 conn(0x7f0f740f0b70 msgr2=0x7f0f740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:34.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.712+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/1354509262 shutdown_connections 2023-12-21T03:43:34.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:34.712+0000 7f0f7b6ce700 1 -- 172.21.15.148:0/1354509262 wait complete. 2023-12-21T03:43:34.714 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:35.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:34.439860+0000 mgr.smithi078.zidbgq (mgr.14162) 129 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:35.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:34.710722+0000 mon.smithi078 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.148:0/1354509262' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:36.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:36.099 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:36.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:36 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:36.440240+0000 mgr.smithi078.zidbgq (mgr.14162) 130 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:37.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.153+0000 7fbfb759e700 1 -- 172.21.15.148:0/1265562608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfb80f62a0 msgr2=0x7fbfb80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:37.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.153+0000 7fbfb759e700 1 --2- 172.21.15.148:0/1265562608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfb80f62a0 0x7fbfb80f66c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fbfac009fb0 tx=0x7fbfac009b30).stop 2023-12-21T03:43:37.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.153+0000 7fbfb759e700 1 -- 172.21.15.148:0/1265562608 shutdown_connections 2023-12-21T03:43:37.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.153+0000 7fbfb759e700 1 --2- 172.21.15.148:0/1265562608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfb80f62a0 0x7fbfb80f66c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:37.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.153+0000 7fbfb759e700 1 -- 172.21.15.148:0/1265562608 >> 172.21.15.148:0/1265562608 conn(0x7fbfb80f1aa0 msgr2=0x7fbfb80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.153+0000 7fbfb759e700 1 -- 172.21.15.148:0/1265562608 shutdown_connections 2023-12-21T03:43:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.153+0000 7fbfb759e700 1 -- 172.21.15.148:0/1265562608 wait complete. 2023-12-21T03:43:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.153+0000 7fbfb759e700 1 Processor -- start 2023-12-21T03:43:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.153+0000 7fbfb759e700 1 -- start start 2023-12-21T03:43:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.153+0000 7fbfb759e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfb80f62a0 0x7fbfb81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.153+0000 7fbfb759e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb81069a0 con 0x7fbfb80f62a0 2023-12-21T03:43:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.154+0000 7fbfb659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfb80f62a0 0x7fbfb81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.154+0000 7fbfb659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfb80f62a0 0x7fbfb81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:55578/0 (socket says 172.21.15.148:55578) 2023-12-21T03:43:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.154+0000 7fbfb659c700 1 -- 172.21.15.148:0/3574735420 learned_addr learned my addr 172.21.15.148:0/3574735420 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.154+0000 7fbfb659c700 1 -- 172.21.15.148:0/3574735420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac005040 con 0x7fbfb80f62a0 2023-12-21T03:43:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.154+0000 7fbfb659c700 1 --2- 172.21.15.148:0/3574735420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfb80f62a0 0x7fbfb81064b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fbfac013510 tx=0x7fbfac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.154+0000 7fbfa6ffd700 1 -- 172.21.15.148:0/3574735420 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfac009320 con 0x7fbfb80f62a0 2023-12-21T03:43:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.154+0000 7fbfa6ffd700 1 -- 172.21.15.148:0/3574735420 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfac013f00 con 0x7fbfb80f62a0 2023-12-21T03:43:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.154+0000 7fbfb759e700 1 -- 172.21.15.148:0/3574735420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb8106ba0 con 0x7fbfb80f62a0 2023-12-21T03:43:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.154+0000 7fbfa6ffd700 1 -- 172.21.15.148:0/3574735420 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfac014dc0 con 0x7fbfb80f62a0 2023-12-21T03:43:37.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.154+0000 7fbfb759e700 1 -- 172.21.15.148:0/3574735420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb8107030 con 0x7fbfb80f62a0 2023-12-21T03:43:37.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.155+0000 7fbfa6ffd700 1 -- 172.21.15.148:0/3574735420 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbfac0095f0 con 0x7fbfb80f62a0 2023-12-21T03:43:37.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.155+0000 7fbfb759e700 1 -- 172.21.15.148:0/3574735420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb8045ad0 con 0x7fbfb80f62a0 2023-12-21T03:43:37.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.155+0000 7fbfa6ffd700 1 --2- 172.21.15.148:0/3574735420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbfa002ff60 0x7fbfa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:37.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.155+0000 7fbfa6ffd700 1 -- 172.21.15.148:0/3574735420 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbfac041040 con 0x7fbfb80f62a0 2023-12-21T03:43:37.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.156+0000 7fbfb5d9b700 1 --2- 172.21.15.148:0/3574735420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbfa002ff60 0x7fbfa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:37.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.156+0000 7fbfb5d9b700 1 --2- 172.21.15.148:0/3574735420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbfa002ff60 0x7fbfa0032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fbfa80096a0 tx=0x7fbfa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:37.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.160+0000 7fbfa6ffd700 1 -- 172.21.15.148:0/3574735420 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbfac014420 con 0x7fbfb80f62a0 2023-12-21T03:43:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.369+0000 7fbfb759e700 1 -- 172.21.15.148:0/3574735420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfb8107ee0 con 0x7fbfb80f62a0 2023-12-21T03:43:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.370+0000 7fbfa6ffd700 1 -- 172.21.15.148:0/3574735420 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbfac010230 con 0x7fbfb80f62a0 2023-12-21T03:43:37.371 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:37.371 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.371+0000 7fbfb759e700 1 -- 172.21.15.148:0/3574735420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbfa002ff60 msgr2=0x7fbfa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.371+0000 7fbfb759e700 1 --2- 172.21.15.148:0/3574735420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbfa002ff60 0x7fbfa0032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fbfa80096a0 tx=0x7fbfa8006b40).stop 2023-12-21T03:43:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.371+0000 7fbfb759e700 1 -- 172.21.15.148:0/3574735420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfb80f62a0 msgr2=0x7fbfb81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.371+0000 7fbfb759e700 1 --2- 172.21.15.148:0/3574735420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfb80f62a0 0x7fbfb81064b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fbfac013510 tx=0x7fbfac00b040).stop 2023-12-21T03:43:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.371+0000 7fbfb759e700 1 -- 172.21.15.148:0/3574735420 shutdown_connections 2023-12-21T03:43:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.371+0000 7fbfb759e700 1 --2- 172.21.15.148:0/3574735420 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbfa002ff60 0x7fbfa0032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.371+0000 7fbfb759e700 1 --2- 172.21.15.148:0/3574735420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfb80f62a0 0x7fbfb81064b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.371+0000 7fbfb759e700 1 -- 172.21.15.148:0/3574735420 >> 172.21.15.148:0/3574735420 conn(0x7fbfb80f1aa0 msgr2=0x7fbfb80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.371+0000 7fbfb759e700 1 -- 172.21.15.148:0/3574735420 shutdown_connections 2023-12-21T03:43:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:37.371+0000 7fbfb759e700 1 -- 172.21.15.148:0/3574735420 wait complete. 2023-12-21T03:43:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:37.370525+0000 mon.smithi078 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.148:0/3574735420' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:38.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:38.767 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:38.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:38.440468+0000 mgr.smithi078.zidbgq (mgr.14162) 131 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:39.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.903+0000 7fcf45819700 1 -- 172.21.15.148:0/2736934225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf400f5410 msgr2=0x7fcf400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:39.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.903+0000 7fcf45819700 1 --2- 172.21.15.148:0/2736934225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf400f5410 0x7fcf400f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fcf28009fb0 tx=0x7fcf28009b30).stop 2023-12-21T03:43:39.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.903+0000 7fcf45819700 1 -- 172.21.15.148:0/2736934225 shutdown_connections 2023-12-21T03:43:39.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.903+0000 7fcf45819700 1 --2- 172.21.15.148:0/2736934225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf400f5410 0x7fcf400f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:39.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.903+0000 7fcf45819700 1 -- 172.21.15.148:0/2736934225 >> 172.21.15.148:0/2736934225 conn(0x7fcf400f0b70 msgr2=0x7fcf400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:39.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.903+0000 7fcf45819700 1 -- 172.21.15.148:0/2736934225 shutdown_connections 2023-12-21T03:43:39.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.903+0000 7fcf45819700 1 -- 172.21.15.148:0/2736934225 wait complete. 2023-12-21T03:43:39.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.904+0000 7fcf45819700 1 Processor -- start 2023-12-21T03:43:39.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.904+0000 7fcf45819700 1 -- start start 2023-12-21T03:43:39.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.904+0000 7fcf45819700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf400f5410 0x7fcf40106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:39.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.904+0000 7fcf45819700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf40106c20 con 0x7fcf400f5410 2023-12-21T03:43:39.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.904+0000 7fcf3effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf400f5410 0x7fcf40106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:39.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.904+0000 7fcf3effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf400f5410 0x7fcf40106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46008/0 (socket says 172.21.15.148:46008) 2023-12-21T03:43:39.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.904+0000 7fcf3effd700 1 -- 172.21.15.148:0/1328221457 learned_addr learned my addr 172.21.15.148:0/1328221457 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:39.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.904+0000 7fcf3effd700 1 -- 172.21.15.148:0/1328221457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf28005040 con 0x7fcf400f5410 2023-12-21T03:43:39.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.904+0000 7fcf3effd700 1 --2- 172.21.15.148:0/1328221457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf400f5410 0x7fcf40106730 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fcf28013510 tx=0x7fcf2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:39.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.905+0000 7fcf377fe700 1 -- 172.21.15.148:0/1328221457 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf28009320 con 0x7fcf400f5410 2023-12-21T03:43:39.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.905+0000 7fcf377fe700 1 -- 172.21.15.148:0/1328221457 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf28013f00 con 0x7fcf400f5410 2023-12-21T03:43:39.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.905+0000 7fcf377fe700 1 -- 172.21.15.148:0/1328221457 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf28014d90 con 0x7fcf400f5410 2023-12-21T03:43:39.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.905+0000 7fcf45819700 1 -- 172.21.15.148:0/1328221457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf40106e20 con 0x7fcf400f5410 2023-12-21T03:43:39.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.905+0000 7fcf45819700 1 -- 172.21.15.148:0/1328221457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf401072b0 con 0x7fcf400f5410 2023-12-21T03:43:39.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.905+0000 7fcf377fe700 1 -- 172.21.15.148:0/1328221457 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf280095f0 con 0x7fcf400f5410 2023-12-21T03:43:39.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.905+0000 7fcf45819700 1 -- 172.21.15.148:0/1328221457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf40045ad0 con 0x7fcf400f5410 2023-12-21T03:43:39.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.906+0000 7fcf377fe700 1 --2- 172.21.15.148:0/1328221457 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcf2c02ff60 0x7fcf2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:39.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.906+0000 7fcf377fe700 1 -- 172.21.15.148:0/1328221457 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf28041000 con 0x7fcf400f5410 2023-12-21T03:43:39.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.906+0000 7fcf3e7fc700 1 --2- 172.21.15.148:0/1328221457 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcf2c02ff60 0x7fcf2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:39.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.906+0000 7fcf3e7fc700 1 --2- 172.21.15.148:0/1328221457 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcf2c02ff60 0x7fcf2c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fcf300096a0 tx=0x7fcf30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:39.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:39.910+0000 7fcf377fe700 1 -- 172.21.15.148:0/1328221457 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf2801d7f0 con 0x7fcf400f5410 2023-12-21T03:43:40.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.115+0000 7fcf45819700 1 -- 172.21.15.148:0/1328221457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf400fae60 con 0x7fcf400f5410 2023-12-21T03:43:40.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.115+0000 7fcf377fe700 1 -- 172.21.15.148:0/1328221457 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf28010230 con 0x7fcf400f5410 2023-12-21T03:43:40.117 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:40.117 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:40.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.117+0000 7fcf45819700 1 -- 172.21.15.148:0/1328221457 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcf2c02ff60 msgr2=0x7fcf2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:40.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.117+0000 7fcf45819700 1 --2- 172.21.15.148:0/1328221457 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcf2c02ff60 0x7fcf2c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fcf300096a0 tx=0x7fcf30006b40).stop 2023-12-21T03:43:40.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.117+0000 7fcf45819700 1 -- 172.21.15.148:0/1328221457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf400f5410 msgr2=0x7fcf40106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:40.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.117+0000 7fcf45819700 1 --2- 172.21.15.148:0/1328221457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf400f5410 0x7fcf40106730 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fcf28013510 tx=0x7fcf2800b040).stop 2023-12-21T03:43:40.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.117+0000 7fcf45819700 1 -- 172.21.15.148:0/1328221457 shutdown_connections 2023-12-21T03:43:40.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.117+0000 7fcf45819700 1 --2- 172.21.15.148:0/1328221457 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcf2c02ff60 0x7fcf2c032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:40.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.117+0000 7fcf45819700 1 --2- 172.21.15.148:0/1328221457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf400f5410 0x7fcf40106730 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:40.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.117+0000 7fcf45819700 1 -- 172.21.15.148:0/1328221457 >> 172.21.15.148:0/1328221457 conn(0x7fcf400f0b70 msgr2=0x7fcf400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:40.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.117+0000 7fcf45819700 1 -- 172.21.15.148:0/1328221457 shutdown_connections 2023-12-21T03:43:40.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:40.117+0000 7fcf45819700 1 -- 172.21.15.148:0/1328221457 wait complete. 2023-12-21T03:43:40.119 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:40.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:40.116364+0000 mon.smithi078 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.148:0/1328221457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:41.488 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:41.488 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:41.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:40.440723+0000 mgr.smithi078.zidbgq (mgr.14162) 132 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:42.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.644+0000 7f0e46c57700 1 -- 172.21.15.148:0/2533451930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e400f5410 msgr2=0x7f0e400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:42.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.644+0000 7f0e46c57700 1 --2- 172.21.15.148:0/2533451930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e400f5410 0x7f0e400f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f0e34009fb0 tx=0x7f0e34009b30).stop 2023-12-21T03:43:42.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.644+0000 7f0e46c57700 1 -- 172.21.15.148:0/2533451930 shutdown_connections 2023-12-21T03:43:42.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.644+0000 7f0e46c57700 1 --2- 172.21.15.148:0/2533451930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e400f5410 0x7f0e400f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:42.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.644+0000 7f0e46c57700 1 -- 172.21.15.148:0/2533451930 >> 172.21.15.148:0/2533451930 conn(0x7f0e400f0b70 msgr2=0x7f0e400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:42.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.644+0000 7f0e46c57700 1 -- 172.21.15.148:0/2533451930 shutdown_connections 2023-12-21T03:43:42.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.644+0000 7f0e46c57700 1 -- 172.21.15.148:0/2533451930 wait complete. 2023-12-21T03:43:42.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.645+0000 7f0e46c57700 1 Processor -- start 2023-12-21T03:43:42.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.645+0000 7f0e46c57700 1 -- start start 2023-12-21T03:43:42.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.645+0000 7f0e46c57700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e400f5410 0x7f0e40109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:42.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.645+0000 7f0e46c57700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e40060220 con 0x7f0e400f5410 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.646+0000 7f0e449f3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e400f5410 0x7f0e40109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.646+0000 7f0e449f3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e400f5410 0x7f0e40109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46026/0 (socket says 172.21.15.148:46026) 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.646+0000 7f0e449f3700 1 -- 172.21.15.148:0/3363953247 learned_addr learned my addr 172.21.15.148:0/3363953247 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.646+0000 7f0e449f3700 1 -- 172.21.15.148:0/3363953247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e34005040 con 0x7f0e400f5410 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.646+0000 7f0e449f3700 1 --2- 172.21.15.148:0/3363953247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e400f5410 0x7f0e40109450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0e34013510 tx=0x7f0e3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.646+0000 7f0e3d7fa700 1 -- 172.21.15.148:0/3363953247 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e34009320 con 0x7f0e400f5410 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.646+0000 7f0e46c57700 1 -- 172.21.15.148:0/3363953247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e40060420 con 0x7f0e400f5410 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.646+0000 7f0e3d7fa700 1 -- 172.21.15.148:0/3363953247 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e34013f00 con 0x7f0e400f5410 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.646+0000 7f0e3d7fa700 1 -- 172.21.15.148:0/3363953247 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e34014d90 con 0x7f0e400f5410 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.646+0000 7f0e46c57700 1 -- 172.21.15.148:0/3363953247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e40109b00 con 0x7f0e400f5410 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.647+0000 7f0e3d7fa700 1 -- 172.21.15.148:0/3363953247 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e34014420 con 0x7f0e400f5410 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.647+0000 7f0e46c57700 1 -- 172.21.15.148:0/3363953247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e40059100 con 0x7f0e400f5410 2023-12-21T03:43:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.647+0000 7f0e3d7fa700 1 --2- 172.21.15.148:0/3363953247 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0e3002ff60 0x7f0e30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:42.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.647+0000 7f0e3d7fa700 1 -- 172.21.15.148:0/3363953247 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e34041000 con 0x7f0e400f5410 2023-12-21T03:43:42.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.648+0000 7f0e3ffff700 1 --2- 172.21.15.148:0/3363953247 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0e3002ff60 0x7f0e30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:42.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.648+0000 7f0e3ffff700 1 --2- 172.21.15.148:0/3363953247 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0e3002ff60 0x7f0e30032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f0e2c0096a0 tx=0x7f0e2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:42.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.652+0000 7f0e3d7fa700 1 -- 172.21.15.148:0/3363953247 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e340095f0 con 0x7f0e400f5410 2023-12-21T03:43:42.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:42.441057+0000 mgr.smithi078.zidbgq (mgr.14162) 133 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:42.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.857+0000 7f0e46c57700 1 -- 172.21.15.148:0/3363953247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e40045ad0 con 0x7f0e400f5410 2023-12-21T03:43:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.858+0000 7f0e3d7fa700 1 -- 172.21.15.148:0/3363953247 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e3401db70 con 0x7f0e400f5410 2023-12-21T03:43:42.859 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:42.859 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.859+0000 7f0e46c57700 1 -- 172.21.15.148:0/3363953247 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0e3002ff60 msgr2=0x7f0e30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.859+0000 7f0e46c57700 1 --2- 172.21.15.148:0/3363953247 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0e3002ff60 0x7f0e30032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f0e2c0096a0 tx=0x7f0e2c006b40).stop 2023-12-21T03:43:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.859+0000 7f0e46c57700 1 -- 172.21.15.148:0/3363953247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e400f5410 msgr2=0x7f0e40109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.859+0000 7f0e46c57700 1 --2- 172.21.15.148:0/3363953247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e400f5410 0x7f0e40109450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0e34013510 tx=0x7f0e3400b040).stop 2023-12-21T03:43:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.859+0000 7f0e46c57700 1 -- 172.21.15.148:0/3363953247 shutdown_connections 2023-12-21T03:43:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.859+0000 7f0e46c57700 1 --2- 172.21.15.148:0/3363953247 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0e3002ff60 0x7f0e30032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.859+0000 7f0e46c57700 1 --2- 172.21.15.148:0/3363953247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e400f5410 0x7f0e40109450 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.859+0000 7f0e46c57700 1 -- 172.21.15.148:0/3363953247 >> 172.21.15.148:0/3363953247 conn(0x7f0e400f0b70 msgr2=0x7f0e400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:42.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.859+0000 7f0e46c57700 1 -- 172.21.15.148:0/3363953247 shutdown_connections 2023-12-21T03:43:42.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:42.859+0000 7f0e46c57700 1 -- 172.21.15.148:0/3363953247 wait complete. 2023-12-21T03:43:42.862 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:42.858682+0000 mon.smithi078 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.148:0/3363953247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:44.223 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:44.223 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:45.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.273+0000 7f242a0cf700 1 -- 172.21.15.148:0/640703255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24240f51a0 msgr2=0x7f24240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:45.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.273+0000 7f242a0cf700 1 --2- 172.21.15.148:0/640703255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24240f51a0 0x7f24240f55c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f2414009fb0 tx=0x7f2414009b30).stop 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.273+0000 7f242a0cf700 1 -- 172.21.15.148:0/640703255 shutdown_connections 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.273+0000 7f242a0cf700 1 --2- 172.21.15.148:0/640703255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24240f51a0 0x7f24240f55c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.273+0000 7f242a0cf700 1 -- 172.21.15.148:0/640703255 >> 172.21.15.148:0/640703255 conn(0x7f24240f09a0 msgr2=0x7f24240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.274+0000 7f242a0cf700 1 -- 172.21.15.148:0/640703255 shutdown_connections 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.274+0000 7f242a0cf700 1 -- 172.21.15.148:0/640703255 wait complete. 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.274+0000 7f242a0cf700 1 Processor -- start 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.274+0000 7f242a0cf700 1 -- start start 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.274+0000 7f242a0cf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24240f51a0 0x7f24241053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.274+0000 7f242a0cf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24241058a0 con 0x7f24240f51a0 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.275+0000 7f24290cd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24240f51a0 0x7f24241053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.275+0000 7f24290cd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24240f51a0 0x7f24241053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46032/0 (socket says 172.21.15.148:46032) 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.275+0000 7f24290cd700 1 -- 172.21.15.148:0/1390828370 learned_addr learned my addr 172.21.15.148:0/1390828370 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.275+0000 7f24290cd700 1 -- 172.21.15.148:0/1390828370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2414005040 con 0x7f24240f51a0 2023-12-21T03:43:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.275+0000 7f24290cd700 1 --2- 172.21.15.148:0/1390828370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24240f51a0 0x7f24241053b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f2414013510 tx=0x7f241400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:45.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.275+0000 7f2419ffb700 1 -- 172.21.15.148:0/1390828370 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2414009320 con 0x7f24240f51a0 2023-12-21T03:43:45.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.275+0000 7f242a0cf700 1 -- 172.21.15.148:0/1390828370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2424105aa0 con 0x7f24240f51a0 2023-12-21T03:43:45.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.275+0000 7f2419ffb700 1 -- 172.21.15.148:0/1390828370 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2414013f00 con 0x7f24240f51a0 2023-12-21T03:43:45.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.275+0000 7f2419ffb700 1 -- 172.21.15.148:0/1390828370 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2414014dc0 con 0x7f24240f51a0 2023-12-21T03:43:45.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.276+0000 7f242a0cf700 1 -- 172.21.15.148:0/1390828370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2424105f30 con 0x7f24240f51a0 2023-12-21T03:43:45.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.276+0000 7f2419ffb700 1 -- 172.21.15.148:0/1390828370 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24140095f0 con 0x7f24240f51a0 2023-12-21T03:43:45.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.276+0000 7f242a0cf700 1 -- 172.21.15.148:0/1390828370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2424045ad0 con 0x7f24240f51a0 2023-12-21T03:43:45.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.276+0000 7f2419ffb700 1 --2- 172.21.15.148:0/1390828370 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f241002ff60 0x7f2410032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:45.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.277+0000 7f2419ffb700 1 -- 172.21.15.148:0/1390828370 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2414042dd0 con 0x7f24240f51a0 2023-12-21T03:43:45.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.277+0000 7f24288cc700 1 --2- 172.21.15.148:0/1390828370 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f241002ff60 0x7f2410032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:45.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.277+0000 7f24288cc700 1 --2- 172.21.15.148:0/1390828370 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f241002ff60 0x7f2410032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f24200096a0 tx=0x7f2420006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:45.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.281+0000 7f2419ffb700 1 -- 172.21.15.148:0/1390828370 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2414014420 con 0x7f24240f51a0 2023-12-21T03:43:45.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.488+0000 7f242a0cf700 1 -- 172.21.15.148:0/1390828370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2424106de0 con 0x7f24240f51a0 2023-12-21T03:43:45.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.489+0000 7f2419ffb700 1 -- 172.21.15.148:0/1390828370 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2414011290 con 0x7f24240f51a0 2023-12-21T03:43:45.491 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:45.491 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:45.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.491+0000 7f242a0cf700 1 -- 172.21.15.148:0/1390828370 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f241002ff60 msgr2=0x7f2410032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:45.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.491+0000 7f242a0cf700 1 --2- 172.21.15.148:0/1390828370 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f241002ff60 0x7f2410032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f24200096a0 tx=0x7f2420006b40).stop 2023-12-21T03:43:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.491+0000 7f242a0cf700 1 -- 172.21.15.148:0/1390828370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24240f51a0 msgr2=0x7f24241053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.491+0000 7f242a0cf700 1 --2- 172.21.15.148:0/1390828370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24240f51a0 0x7f24241053b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f2414013510 tx=0x7f241400b040).stop 2023-12-21T03:43:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.491+0000 7f242a0cf700 1 -- 172.21.15.148:0/1390828370 shutdown_connections 2023-12-21T03:43:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.491+0000 7f242a0cf700 1 --2- 172.21.15.148:0/1390828370 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f241002ff60 0x7f2410032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.491+0000 7f242a0cf700 1 --2- 172.21.15.148:0/1390828370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24240f51a0 0x7f24241053b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.491+0000 7f242a0cf700 1 -- 172.21.15.148:0/1390828370 >> 172.21.15.148:0/1390828370 conn(0x7f24240f09a0 msgr2=0x7f24240f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.491+0000 7f242a0cf700 1 -- 172.21.15.148:0/1390828370 shutdown_connections 2023-12-21T03:43:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:45.491+0000 7f242a0cf700 1 -- 172.21.15.148:0/1390828370 wait complete. 2023-12-21T03:43:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:45.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:44.441359+0000 mgr.smithi078.zidbgq (mgr.14162) 134 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:46.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:46 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:45.490697+0000 mon.smithi078 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.148:0/1390828370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:46.886 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:46.887 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:47.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:46.441614+0000 mgr.smithi078.zidbgq (mgr.14162) 135 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:48.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.997+0000 7f84ee45c700 1 -- 172.21.15.148:0/2391007587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e80f51a0 msgr2=0x7f84e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:48.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.997+0000 7f84ee45c700 1 --2- 172.21.15.148:0/2391007587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e80f51a0 0x7f84e80f55c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f84d8009fb0 tx=0x7f84d8009b30).stop 2023-12-21T03:43:48.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.997+0000 7f84ee45c700 1 -- 172.21.15.148:0/2391007587 shutdown_connections 2023-12-21T03:43:48.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.997+0000 7f84ee45c700 1 --2- 172.21.15.148:0/2391007587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e80f51a0 0x7f84e80f55c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:48.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.997+0000 7f84ee45c700 1 -- 172.21.15.148:0/2391007587 >> 172.21.15.148:0/2391007587 conn(0x7f84e80f09a0 msgr2=0x7f84e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:48.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.997+0000 7f84ee45c700 1 -- 172.21.15.148:0/2391007587 shutdown_connections 2023-12-21T03:43:48.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.997+0000 7f84ee45c700 1 -- 172.21.15.148:0/2391007587 wait complete. 2023-12-21T03:43:48.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.997+0000 7f84ee45c700 1 Processor -- start 2023-12-21T03:43:48.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.997+0000 7f84ee45c700 1 -- start start 2023-12-21T03:43:48.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.998+0000 7f84ee45c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e80f51a0 0x7f84e81091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:48.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.998+0000 7f84ee45c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8060240 con 0x7f84e80f51a0 2023-12-21T03:43:48.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.998+0000 7f84ed45a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e80f51a0 0x7f84e81091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:48.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.998+0000 7f84ed45a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e80f51a0 0x7f84e81091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46044/0 (socket says 172.21.15.148:46044) 2023-12-21T03:43:48.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.998+0000 7f84ed45a700 1 -- 172.21.15.148:0/4035310697 learned_addr learned my addr 172.21.15.148:0/4035310697 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:48.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.998+0000 7f84ed45a700 1 -- 172.21.15.148:0/4035310697 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d8005040 con 0x7f84e80f51a0 2023-12-21T03:43:48.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.998+0000 7f84ed45a700 1 --2- 172.21.15.148:0/4035310697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e80f51a0 0x7f84e81091b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f84d8009830 tx=0x7f84d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:48.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.998+0000 7f84ddffb700 1 -- 172.21.15.148:0/4035310697 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84d8009320 con 0x7f84e80f51a0 2023-12-21T03:43:48.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.998+0000 7f84ee45c700 1 -- 172.21.15.148:0/4035310697 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e8060440 con 0x7f84e80f51a0 2023-12-21T03:43:48.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.998+0000 7f84ddffb700 1 -- 172.21.15.148:0/4035310697 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84d80136c0 con 0x7f84e80f51a0 2023-12-21T03:43:48.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.998+0000 7f84ddffb700 1 -- 172.21.15.148:0/4035310697 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84d801ee90 con 0x7f84e80f51a0 2023-12-21T03:43:48.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.999+0000 7f84ee45c700 1 -- 172.21.15.148:0/4035310697 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e8109860 con 0x7f84e80f51a0 2023-12-21T03:43:48.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.999+0000 7f84ddffb700 1 -- 172.21.15.148:0/4035310697 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84d801e420 con 0x7f84e80f51a0 2023-12-21T03:43:48.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:47.999+0000 7f84ee45c700 1 -- 172.21.15.148:0/4035310697 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e8045ad0 con 0x7f84e80f51a0 2023-12-21T03:43:48.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.000+0000 7f84ddffb700 1 --2- 172.21.15.148:0/4035310697 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f84d402ff10 0x7f84d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:48.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.000+0000 7f84ddffb700 1 -- 172.21.15.148:0/4035310697 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84d8040fa0 con 0x7f84e80f51a0 2023-12-21T03:43:48.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.000+0000 7f84ecc59700 1 --2- 172.21.15.148:0/4035310697 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f84d402ff10 0x7f84d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:48.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.000+0000 7f84ecc59700 1 --2- 172.21.15.148:0/4035310697 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f84d402ff10 0x7f84d40323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f84e40096a0 tx=0x7f84e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:48.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.004+0000 7f84ddffb700 1 -- 172.21.15.148:0/4035310697 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84d80095f0 con 0x7f84e80f51a0 2023-12-21T03:43:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.213+0000 7f84ee45c700 1 -- 172.21.15.148:0/4035310697 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84e810a070 con 0x7f84e80f51a0 2023-12-21T03:43:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.213+0000 7f84ddffb700 1 -- 172.21.15.148:0/4035310697 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f84d802c2d0 con 0x7f84e80f51a0 2023-12-21T03:43:48.215 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:48.216 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.215+0000 7f84ee45c700 1 -- 172.21.15.148:0/4035310697 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f84d402ff10 msgr2=0x7f84d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.215+0000 7f84ee45c700 1 --2- 172.21.15.148:0/4035310697 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f84d402ff10 0x7f84d40323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f84e40096a0 tx=0x7f84e4006b40).stop 2023-12-21T03:43:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.215+0000 7f84ee45c700 1 -- 172.21.15.148:0/4035310697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e80f51a0 msgr2=0x7f84e81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.215+0000 7f84ee45c700 1 --2- 172.21.15.148:0/4035310697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e80f51a0 0x7f84e81091b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f84d8009830 tx=0x7f84d800b040).stop 2023-12-21T03:43:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.215+0000 7f84ee45c700 1 -- 172.21.15.148:0/4035310697 shutdown_connections 2023-12-21T03:43:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.215+0000 7f84ee45c700 1 --2- 172.21.15.148:0/4035310697 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f84d402ff10 0x7f84d40323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.215+0000 7f84ee45c700 1 --2- 172.21.15.148:0/4035310697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e80f51a0 0x7f84e81091b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.215+0000 7f84ee45c700 1 -- 172.21.15.148:0/4035310697 >> 172.21.15.148:0/4035310697 conn(0x7f84e80f09a0 msgr2=0x7f84e80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.215+0000 7f84ee45c700 1 -- 172.21.15.148:0/4035310697 shutdown_connections 2023-12-21T03:43:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:48.215+0000 7f84ee45c700 1 -- 172.21.15.148:0/4035310697 wait complete. 2023-12-21T03:43:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:48.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:48.215222+0000 mon.smithi078 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.148:0/4035310697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:49.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:48.441970+0000 mgr.smithi078.zidbgq (mgr.14162) 136 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:49.615 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:49.616 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:50.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.715+0000 7f811c1fe700 1 -- 172.21.15.148:0/193122761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81140f5410 msgr2=0x7f81140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:50.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.715+0000 7f811c1fe700 1 --2- 172.21.15.148:0/193122761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81140f5410 0x7f81140f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f8108009fb0 tx=0x7f8108009b30).stop 2023-12-21T03:43:50.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.715+0000 7f811c1fe700 1 -- 172.21.15.148:0/193122761 shutdown_connections 2023-12-21T03:43:50.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.715+0000 7f811c1fe700 1 --2- 172.21.15.148:0/193122761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81140f5410 0x7f81140f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:50.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.715+0000 7f811c1fe700 1 -- 172.21.15.148:0/193122761 >> 172.21.15.148:0/193122761 conn(0x7f81140f0b70 msgr2=0x7f81140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:50.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.716+0000 7f811c1fe700 1 -- 172.21.15.148:0/193122761 shutdown_connections 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.716+0000 7f811c1fe700 1 -- 172.21.15.148:0/193122761 wait complete. 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.716+0000 7f811c1fe700 1 Processor -- start 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.716+0000 7f811c1fe700 1 -- start start 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.716+0000 7f811c1fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81140f5410 0x7f8114105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.716+0000 7f811c1fe700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8114105b20 con 0x7f81140f5410 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.717+0000 7f8119f9a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81140f5410 0x7f8114105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.717+0000 7f8119f9a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81140f5410 0x7f8114105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35912/0 (socket says 172.21.15.148:35912) 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.717+0000 7f8119f9a700 1 -- 172.21.15.148:0/532901293 learned_addr learned my addr 172.21.15.148:0/532901293 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.717+0000 7f8119f9a700 1 -- 172.21.15.148:0/532901293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8108005040 con 0x7f81140f5410 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.717+0000 7f8119f9a700 1 --2- 172.21.15.148:0/532901293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81140f5410 0x7f8114105630 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f8108013510 tx=0x7f810800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.717+0000 7f81067fc700 1 -- 172.21.15.148:0/532901293 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8108009320 con 0x7f81140f5410 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.717+0000 7f811c1fe700 1 -- 172.21.15.148:0/532901293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8114105d20 con 0x7f81140f5410 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.717+0000 7f81067fc700 1 -- 172.21.15.148:0/532901293 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8108013f00 con 0x7f81140f5410 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.717+0000 7f81067fc700 1 -- 172.21.15.148:0/532901293 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8108014d90 con 0x7f81140f5410 2023-12-21T03:43:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.717+0000 7f811c1fe700 1 -- 172.21.15.148:0/532901293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81141061b0 con 0x7f81140f5410 2023-12-21T03:43:50.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.718+0000 7f811c1fe700 1 -- 172.21.15.148:0/532901293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8114045ad0 con 0x7f81140f5410 2023-12-21T03:43:50.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.718+0000 7f81067fc700 1 -- 172.21.15.148:0/532901293 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81080095f0 con 0x7f81140f5410 2023-12-21T03:43:50.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.718+0000 7f81067fc700 1 --2- 172.21.15.148:0/532901293 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f810002ff60 0x7f8100032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:50.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.718+0000 7f81067fc700 1 -- 172.21.15.148:0/532901293 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8108040fc0 con 0x7f81140f5410 2023-12-21T03:43:50.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.718+0000 7f8119799700 1 --2- 172.21.15.148:0/532901293 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f810002ff60 0x7f8100032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:50.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.719+0000 7f8119799700 1 --2- 172.21.15.148:0/532901293 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f810002ff60 0x7f8100032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f81100096a0 tx=0x7f8110006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:50.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.723+0000 7f81067fc700 1 -- 172.21.15.148:0/532901293 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8108014420 con 0x7f81140f5410 2023-12-21T03:43:50.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:50.442198+0000 mgr.smithi078.zidbgq (mgr.14162) 137 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:50.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.928+0000 7f811c1fe700 1 -- 172.21.15.148:0/532901293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81140fae60 con 0x7f81140f5410 2023-12-21T03:43:50.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.928+0000 7f81067fc700 1 -- 172.21.15.148:0/532901293 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8108010230 con 0x7f81140f5410 2023-12-21T03:43:50.931 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:50.931 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:50.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.930+0000 7f811c1fe700 1 -- 172.21.15.148:0/532901293 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f810002ff60 msgr2=0x7f8100032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:50.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.930+0000 7f811c1fe700 1 --2- 172.21.15.148:0/532901293 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f810002ff60 0x7f8100032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f81100096a0 tx=0x7f8110006b40).stop 2023-12-21T03:43:50.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.930+0000 7f811c1fe700 1 -- 172.21.15.148:0/532901293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81140f5410 msgr2=0x7f8114105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:50.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.930+0000 7f811c1fe700 1 --2- 172.21.15.148:0/532901293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81140f5410 0x7f8114105630 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f8108013510 tx=0x7f810800b040).stop 2023-12-21T03:43:50.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.930+0000 7f811c1fe700 1 -- 172.21.15.148:0/532901293 shutdown_connections 2023-12-21T03:43:50.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.930+0000 7f811c1fe700 1 --2- 172.21.15.148:0/532901293 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f810002ff60 0x7f8100032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:50.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.930+0000 7f811c1fe700 1 --2- 172.21.15.148:0/532901293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81140f5410 0x7f8114105630 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:50.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.930+0000 7f811c1fe700 1 -- 172.21.15.148:0/532901293 >> 172.21.15.148:0/532901293 conn(0x7f81140f0b70 msgr2=0x7f81140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:50.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.930+0000 7f811c1fe700 1 -- 172.21.15.148:0/532901293 shutdown_connections 2023-12-21T03:43:50.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:50.930+0000 7f811c1fe700 1 -- 172.21.15.148:0/532901293 wait complete. 2023-12-21T03:43:50.934 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:50.930154+0000 mon.smithi078 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.148:0/532901293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:52.301 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:52.301 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:52.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:52.442504+0000 mgr.smithi078.zidbgq (mgr.14162) 138 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:53.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.400+0000 7fce1a4bb700 1 -- 172.21.15.148:0/3289667376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce14060200 msgr2=0x7fce140fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:53.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.400+0000 7fce1a4bb700 1 --2- 172.21.15.148:0/3289667376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce14060200 0x7fce140fb570 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fce04009fb0 tx=0x7fce04009b30).stop 2023-12-21T03:43:53.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.400+0000 7fce1a4bb700 1 -- 172.21.15.148:0/3289667376 shutdown_connections 2023-12-21T03:43:53.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.400+0000 7fce1a4bb700 1 --2- 172.21.15.148:0/3289667376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce14060200 0x7fce140fb570 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:53.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.400+0000 7fce1a4bb700 1 -- 172.21.15.148:0/3289667376 >> 172.21.15.148:0/3289667376 conn(0x7fce140f0b70 msgr2=0x7fce140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:53.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.401+0000 7fce1a4bb700 1 -- 172.21.15.148:0/3289667376 shutdown_connections 2023-12-21T03:43:53.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.401+0000 7fce1a4bb700 1 -- 172.21.15.148:0/3289667376 wait complete. 2023-12-21T03:43:53.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.401+0000 7fce1a4bb700 1 Processor -- start 2023-12-21T03:43:53.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.401+0000 7fce1a4bb700 1 -- start start 2023-12-21T03:43:53.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.401+0000 7fce1a4bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce14060200 0x7fce141088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:53.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.401+0000 7fce1a4bb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce14108da0 con 0x7fce14060200 2023-12-21T03:43:53.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.402+0000 7fce13fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce14060200 0x7fce141088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:53.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.402+0000 7fce13fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce14060200 0x7fce141088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35932/0 (socket says 172.21.15.148:35932) 2023-12-21T03:43:53.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.402+0000 7fce13fff700 1 -- 172.21.15.148:0/2101302038 learned_addr learned my addr 172.21.15.148:0/2101302038 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:53.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.403+0000 7fce13fff700 1 -- 172.21.15.148:0/2101302038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce04005040 con 0x7fce14060200 2023-12-21T03:43:53.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.403+0000 7fce13fff700 1 --2- 172.21.15.148:0/2101302038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce14060200 0x7fce141088b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fce04013510 tx=0x7fce0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:53.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.403+0000 7fce10ff9700 1 -- 172.21.15.148:0/2101302038 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce04009320 con 0x7fce14060200 2023-12-21T03:43:53.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.403+0000 7fce10ff9700 1 -- 172.21.15.148:0/2101302038 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce04013f00 con 0x7fce14060200 2023-12-21T03:43:53.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.403+0000 7fce1a4bb700 1 -- 172.21.15.148:0/2101302038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce14108fa0 con 0x7fce14060200 2023-12-21T03:43:53.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.403+0000 7fce10ff9700 1 -- 172.21.15.148:0/2101302038 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce04014dc0 con 0x7fce14060200 2023-12-21T03:43:53.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.403+0000 7fce1a4bb700 1 -- 172.21.15.148:0/2101302038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce14109430 con 0x7fce14060200 2023-12-21T03:43:53.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.404+0000 7fce1a4bb700 1 -- 172.21.15.148:0/2101302038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce14045ad0 con 0x7fce14060200 2023-12-21T03:43:53.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.405+0000 7fce10ff9700 1 -- 172.21.15.148:0/2101302038 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce040095f0 con 0x7fce14060200 2023-12-21T03:43:53.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.405+0000 7fce10ff9700 1 --2- 172.21.15.148:0/2101302038 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdfc030000 0x7fcdfc0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:53.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.405+0000 7fce10ff9700 1 -- 172.21.15.148:0/2101302038 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce04041610 con 0x7fce14060200 2023-12-21T03:43:53.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.406+0000 7fce137fe700 1 --2- 172.21.15.148:0/2101302038 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdfc030000 0x7fcdfc0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:53.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.406+0000 7fce137fe700 1 --2- 172.21.15.148:0/2101302038 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdfc030000 0x7fcdfc0324c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fce080096a0 tx=0x7fce08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:53.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.412+0000 7fce10ff9700 1 -- 172.21.15.148:0/2101302038 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce04014420 con 0x7fce14060200 2023-12-21T03:43:53.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.618+0000 7fce1a4bb700 1 -- 172.21.15.148:0/2101302038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce1410a280 con 0x7fce14060200 2023-12-21T03:43:53.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.619+0000 7fce10ff9700 1 -- 172.21.15.148:0/2101302038 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce0402d090 con 0x7fce14060200 2023-12-21T03:43:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:53.621 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:53.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.620+0000 7fce1a4bb700 1 -- 172.21.15.148:0/2101302038 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdfc030000 msgr2=0x7fcdfc0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:53.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.620+0000 7fce1a4bb700 1 --2- 172.21.15.148:0/2101302038 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdfc030000 0x7fcdfc0324c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fce080096a0 tx=0x7fce08006b40).stop 2023-12-21T03:43:53.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.620+0000 7fce1a4bb700 1 -- 172.21.15.148:0/2101302038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce14060200 msgr2=0x7fce141088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:53.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.620+0000 7fce1a4bb700 1 --2- 172.21.15.148:0/2101302038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce14060200 0x7fce141088b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fce04013510 tx=0x7fce0400b040).stop 2023-12-21T03:43:53.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.620+0000 7fce1a4bb700 1 -- 172.21.15.148:0/2101302038 shutdown_connections 2023-12-21T03:43:53.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.620+0000 7fce1a4bb700 1 --2- 172.21.15.148:0/2101302038 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdfc030000 0x7fcdfc0324c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:53.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.620+0000 7fce1a4bb700 1 --2- 172.21.15.148:0/2101302038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce14060200 0x7fce141088b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:53.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.620+0000 7fce1a4bb700 1 -- 172.21.15.148:0/2101302038 >> 172.21.15.148:0/2101302038 conn(0x7fce140f0b70 msgr2=0x7fce140f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:53.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.620+0000 7fce1a4bb700 1 -- 172.21.15.148:0/2101302038 shutdown_connections 2023-12-21T03:43:53.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:53.620+0000 7fce1a4bb700 1 -- 172.21.15.148:0/2101302038 wait complete. 2023-12-21T03:43:53.623 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:54.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:53.620579+0000 mon.smithi078 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.148:0/2101302038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:55.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:55.052 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:55.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:54.442732+0000 mgr.smithi078.zidbgq (mgr.14162) 139 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:56.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.183+0000 7f50e7834700 1 -- 172.21.15.148:0/4157466665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50e00f5410 msgr2=0x7f50e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:56.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.183+0000 7f50e7834700 1 --2- 172.21.15.148:0/4157466665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50e00f5410 0x7f50e00f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f50d4009fb0 tx=0x7f50d4009b30).stop 2023-12-21T03:43:56.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.183+0000 7f50e7834700 1 -- 172.21.15.148:0/4157466665 shutdown_connections 2023-12-21T03:43:56.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.183+0000 7f50e7834700 1 --2- 172.21.15.148:0/4157466665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50e00f5410 0x7f50e00f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:56.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.183+0000 7f50e7834700 1 -- 172.21.15.148:0/4157466665 >> 172.21.15.148:0/4157466665 conn(0x7f50e00f0b70 msgr2=0x7f50e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:56.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.183+0000 7f50e7834700 1 -- 172.21.15.148:0/4157466665 shutdown_connections 2023-12-21T03:43:56.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.183+0000 7f50e7834700 1 -- 172.21.15.148:0/4157466665 wait complete. 2023-12-21T03:43:56.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.184+0000 7f50e7834700 1 Processor -- start 2023-12-21T03:43:56.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.184+0000 7f50e7834700 1 -- start start 2023-12-21T03:43:56.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.184+0000 7f50e7834700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50e00f5410 0x7f50e00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:56.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.184+0000 7f50e7834700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00fb820 con 0x7f50e00f5410 2023-12-21T03:43:56.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.185+0000 7f50e55d0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50e00f5410 0x7f50e00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.185+0000 7f50e55d0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50e00f5410 0x7f50e00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35938/0 (socket says 172.21.15.148:35938) 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.185+0000 7f50e55d0700 1 -- 172.21.15.148:0/3983950257 learned_addr learned my addr 172.21.15.148:0/3983950257 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.185+0000 7f50e55d0700 1 -- 172.21.15.148:0/3983950257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d4005040 con 0x7f50e00f5410 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.185+0000 7f50e55d0700 1 --2- 172.21.15.148:0/3983950257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50e00f5410 0x7f50e00fd3a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f50d4013510 tx=0x7f50d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.185+0000 7f50d1ffb700 1 -- 172.21.15.148:0/3983950257 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d4009320 con 0x7f50e00f5410 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.185+0000 7f50e7834700 1 -- 172.21.15.148:0/3983950257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e00fb9c0 con 0x7f50e00f5410 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.185+0000 7f50d1ffb700 1 -- 172.21.15.148:0/3983950257 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50d4013f00 con 0x7f50e00f5410 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.185+0000 7f50d1ffb700 1 -- 172.21.15.148:0/3983950257 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d4014d90 con 0x7f50e00f5410 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.185+0000 7f50e7834700 1 -- 172.21.15.148:0/3983950257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e00fbe50 con 0x7f50e00f5410 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.186+0000 7f50d1ffb700 1 -- 172.21.15.148:0/3983950257 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50d40263a0 con 0x7f50e00f5410 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.186+0000 7f50e7834700 1 -- 172.21.15.148:0/3983950257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e0059100 con 0x7f50e00f5410 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.186+0000 7f50d1ffb700 1 --2- 172.21.15.148:0/3983950257 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f50cc02ff60 0x7f50cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.186+0000 7f50d1ffb700 1 -- 172.21.15.148:0/3983950257 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50d40413a0 con 0x7f50e00f5410 2023-12-21T03:43:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.187+0000 7f50e4dcf700 1 --2- 172.21.15.148:0/3983950257 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f50cc02ff60 0x7f50cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:56.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.187+0000 7f50e4dcf700 1 --2- 172.21.15.148:0/3983950257 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f50cc02ff60 0x7f50cc032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f50dc0096a0 tx=0x7f50dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:56.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.191+0000 7f50d1ffb700 1 -- 172.21.15.148:0/3983950257 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50d40126f0 con 0x7f50e00f5410 2023-12-21T03:43:56.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.396+0000 7f50e7834700 1 -- 172.21.15.148:0/3983950257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50e0045ad0 con 0x7f50e00f5410 2023-12-21T03:43:56.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.396+0000 7f50d1ffb700 1 -- 172.21.15.148:0/3983950257 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f50d400f340 con 0x7f50e00f5410 2023-12-21T03:43:56.397 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:56.398 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:56.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.398+0000 7f50e7834700 1 -- 172.21.15.148:0/3983950257 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f50cc02ff60 msgr2=0x7f50cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:56.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.398+0000 7f50e7834700 1 --2- 172.21.15.148:0/3983950257 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f50cc02ff60 0x7f50cc032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f50dc0096a0 tx=0x7f50dc006b40).stop 2023-12-21T03:43:56.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.398+0000 7f50e7834700 1 -- 172.21.15.148:0/3983950257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50e00f5410 msgr2=0x7f50e00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:56.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.398+0000 7f50e7834700 1 --2- 172.21.15.148:0/3983950257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50e00f5410 0x7f50e00fd3a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f50d4013510 tx=0x7f50d400b040).stop 2023-12-21T03:43:56.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.398+0000 7f50e7834700 1 -- 172.21.15.148:0/3983950257 shutdown_connections 2023-12-21T03:43:56.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.398+0000 7f50e7834700 1 --2- 172.21.15.148:0/3983950257 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f50cc02ff60 0x7f50cc032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:56.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.398+0000 7f50e7834700 1 --2- 172.21.15.148:0/3983950257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50e00f5410 0x7f50e00fd3a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:56.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.398+0000 7f50e7834700 1 -- 172.21.15.148:0/3983950257 >> 172.21.15.148:0/3983950257 conn(0x7f50e00f0b70 msgr2=0x7f50e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:56.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.398+0000 7f50e7834700 1 -- 172.21.15.148:0/3983950257 shutdown_connections 2023-12-21T03:43:56.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:56.398+0000 7f50e7834700 1 -- 172.21.15.148:0/3983950257 wait complete. 2023-12-21T03:43:56.400 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:56.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:56.397007+0000 mon.smithi078 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.148:0/3983950257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:43:57.764 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:43:57.765 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:43:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:56.442986+0000 mgr.smithi078.zidbgq (mgr.14162) 140 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:58.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.870+0000 7fc0c72c4700 1 -- 172.21.15.148:0/2040560435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c00f5410 msgr2=0x7fc0c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:58.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.870+0000 7fc0c72c4700 1 --2- 172.21.15.148:0/2040560435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c00f5410 0x7fc0c00f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fc0bc009fb0 tx=0x7fc0bc009b30).stop 2023-12-21T03:43:58.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.870+0000 7fc0c72c4700 1 -- 172.21.15.148:0/2040560435 shutdown_connections 2023-12-21T03:43:58.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.870+0000 7fc0c72c4700 1 --2- 172.21.15.148:0/2040560435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c00f5410 0x7fc0c00f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:58.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.870+0000 7fc0c72c4700 1 -- 172.21.15.148:0/2040560435 >> 172.21.15.148:0/2040560435 conn(0x7fc0c00f0b70 msgr2=0x7fc0c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:58.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.870+0000 7fc0c72c4700 1 -- 172.21.15.148:0/2040560435 shutdown_connections 2023-12-21T03:43:58.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.870+0000 7fc0c72c4700 1 -- 172.21.15.148:0/2040560435 wait complete. 2023-12-21T03:43:58.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.871+0000 7fc0c72c4700 1 Processor -- start 2023-12-21T03:43:58.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.871+0000 7fc0c72c4700 1 -- start start 2023-12-21T03:43:58.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.871+0000 7fc0c72c4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c00f5410 0x7fc0c0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.871+0000 7fc0c72c4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c0060240 con 0x7fc0c00f5410 2023-12-21T03:43:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.871+0000 7fc0c5060700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c00f5410 0x7fc0c0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.871+0000 7fc0c5060700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c00f5410 0x7fc0c0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35958/0 (socket says 172.21.15.148:35958) 2023-12-21T03:43:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.871+0000 7fc0c5060700 1 -- 172.21.15.148:0/1459851205 learned_addr learned my addr 172.21.15.148:0/1459851205 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:43:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.872+0000 7fc0c5060700 1 -- 172.21.15.148:0/1459851205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc005040 con 0x7fc0c00f5410 2023-12-21T03:43:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.872+0000 7fc0c5060700 1 --2- 172.21.15.148:0/1459851205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c00f5410 0x7fc0c0109430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc0bc014000 tx=0x7fc0bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.872+0000 7fc0b5ffb700 1 -- 172.21.15.148:0/1459851205 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0bc0095f0 con 0x7fc0c00f5410 2023-12-21T03:43:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.872+0000 7fc0c72c4700 1 -- 172.21.15.148:0/1459851205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c0060440 con 0x7fc0c00f5410 2023-12-21T03:43:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.872+0000 7fc0b5ffb700 1 -- 172.21.15.148:0/1459851205 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0bc007e70 con 0x7fc0c00f5410 2023-12-21T03:43:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.872+0000 7fc0b5ffb700 1 -- 172.21.15.148:0/1459851205 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0bc014ea0 con 0x7fc0c00f5410 2023-12-21T03:43:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.872+0000 7fc0c72c4700 1 -- 172.21.15.148:0/1459851205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c0109ae0 con 0x7fc0c00f5410 2023-12-21T03:43:58.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.873+0000 7fc0b5ffb700 1 -- 172.21.15.148:0/1459851205 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0bc025070 con 0x7fc0c00f5410 2023-12-21T03:43:58.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.873+0000 7fc0c72c4700 1 -- 172.21.15.148:0/1459851205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c0045ad0 con 0x7fc0c00f5410 2023-12-21T03:43:58.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.873+0000 7fc0b5ffb700 1 --2- 172.21.15.148:0/1459851205 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc0ac02ff60 0x7fc0ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:43:58.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.873+0000 7fc0b5ffb700 1 -- 172.21.15.148:0/1459851205 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0bc0409b0 con 0x7fc0c00f5410 2023-12-21T03:43:58.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.874+0000 7fc0c485f700 1 --2- 172.21.15.148:0/1459851205 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc0ac02ff60 0x7fc0ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:43:58.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.874+0000 7fc0c485f700 1 --2- 172.21.15.148:0/1459851205 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc0ac02ff60 0x7fc0ac032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fc0b00096a0 tx=0x7fc0b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:43:58.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:58.878+0000 7fc0b5ffb700 1 -- 172.21.15.148:0/1459851205 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0bc00f4e0 con 0x7fc0c00f5410 2023-12-21T03:43:59.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.089+0000 7fc0c72c4700 1 -- 172.21.15.148:0/1459851205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0c00fae60 con 0x7fc0c00f5410 2023-12-21T03:43:59.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.089+0000 7fc0b5ffb700 1 -- 172.21.15.148:0/1459851205 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc0bc00f300 con 0x7fc0c00f5410 2023-12-21T03:43:59.091 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:43:59.091 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:43:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.091+0000 7fc0c72c4700 1 -- 172.21.15.148:0/1459851205 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc0ac02ff60 msgr2=0x7fc0ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.091+0000 7fc0c72c4700 1 --2- 172.21.15.148:0/1459851205 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc0ac02ff60 0x7fc0ac032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fc0b00096a0 tx=0x7fc0b0006b40).stop 2023-12-21T03:43:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.091+0000 7fc0c72c4700 1 -- 172.21.15.148:0/1459851205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c00f5410 msgr2=0x7fc0c0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:43:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.091+0000 7fc0c72c4700 1 --2- 172.21.15.148:0/1459851205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c00f5410 0x7fc0c0109430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc0bc014000 tx=0x7fc0bc00b040).stop 2023-12-21T03:43:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.091+0000 7fc0c72c4700 1 -- 172.21.15.148:0/1459851205 shutdown_connections 2023-12-21T03:43:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.091+0000 7fc0c72c4700 1 --2- 172.21.15.148:0/1459851205 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc0ac02ff60 0x7fc0ac032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.091+0000 7fc0c72c4700 1 --2- 172.21.15.148:0/1459851205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c00f5410 0x7fc0c0109430 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:43:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.091+0000 7fc0c72c4700 1 -- 172.21.15.148:0/1459851205 >> 172.21.15.148:0/1459851205 conn(0x7fc0c00f0b70 msgr2=0x7fc0c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:43:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.091+0000 7fc0c72c4700 1 -- 172.21.15.148:0/1459851205 shutdown_connections 2023-12-21T03:43:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:43:59.091+0000 7fc0c72c4700 1 -- 172.21.15.148:0/1459851205 wait complete. 2023-12-21T03:43:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:43:59.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:43:58.443226+0000 mgr.smithi078.zidbgq (mgr.14162) 141 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:43:59.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:43:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:43:59.090368+0000 mon.smithi078 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.148:0/1459851205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:00.465 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:00.465 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:01.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.597+0000 7f4a32328700 1 -- 172.21.15.148:0/3591951405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2c0f5410 msgr2=0x7f4a2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:01.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.597+0000 7f4a32328700 1 --2- 172.21.15.148:0/3591951405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2c0f5410 0x7f4a2c0f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f4a1c009fb0 tx=0x7f4a1c009b30).stop 2023-12-21T03:44:01.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.597+0000 7f4a32328700 1 -- 172.21.15.148:0/3591951405 shutdown_connections 2023-12-21T03:44:01.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.597+0000 7f4a32328700 1 --2- 172.21.15.148:0/3591951405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2c0f5410 0x7f4a2c0f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:01.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.597+0000 7f4a32328700 1 -- 172.21.15.148:0/3591951405 >> 172.21.15.148:0/3591951405 conn(0x7f4a2c0f0b70 msgr2=0x7f4a2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:01.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.597+0000 7f4a32328700 1 -- 172.21.15.148:0/3591951405 shutdown_connections 2023-12-21T03:44:01.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.597+0000 7f4a32328700 1 -- 172.21.15.148:0/3591951405 wait complete. 2023-12-21T03:44:01.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.598+0000 7f4a32328700 1 Processor -- start 2023-12-21T03:44:01.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.598+0000 7f4a32328700 1 -- start start 2023-12-21T03:44:01.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.598+0000 7f4a32328700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2c0f5410 0x7f4a2c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:01.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.598+0000 7f4a32328700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c060220 con 0x7f4a2c0f5410 2023-12-21T03:44:01.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.599+0000 7f4a2bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2c0f5410 0x7f4a2c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:01.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.599+0000 7f4a2bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2c0f5410 0x7f4a2c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:50068/0 (socket says 172.21.15.148:50068) 2023-12-21T03:44:01.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.599+0000 7f4a2bfff700 1 -- 172.21.15.148:0/2422091116 learned_addr learned my addr 172.21.15.148:0/2422091116 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:01.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.600+0000 7f4a2bfff700 1 -- 172.21.15.148:0/2422091116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c005040 con 0x7f4a2c0f5410 2023-12-21T03:44:01.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.600+0000 7f4a2bfff700 1 --2- 172.21.15.148:0/2422091116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2c0f5410 0x7f4a2c109450 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f4a1c013510 tx=0x7f4a1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:01.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.600+0000 7f4a28ff9700 1 -- 172.21.15.148:0/2422091116 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a1c009320 con 0x7f4a2c0f5410 2023-12-21T03:44:01.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.600+0000 7f4a32328700 1 -- 172.21.15.148:0/2422091116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c060420 con 0x7f4a2c0f5410 2023-12-21T03:44:01.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.600+0000 7f4a28ff9700 1 -- 172.21.15.148:0/2422091116 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a1c013f00 con 0x7f4a2c0f5410 2023-12-21T03:44:01.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.600+0000 7f4a28ff9700 1 -- 172.21.15.148:0/2422091116 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a1c014d90 con 0x7f4a2c0f5410 2023-12-21T03:44:01.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.600+0000 7f4a32328700 1 -- 172.21.15.148:0/2422091116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c109b00 con 0x7f4a2c0f5410 2023-12-21T03:44:01.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.601+0000 7f4a28ff9700 1 -- 172.21.15.148:0/2422091116 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a1c0095f0 con 0x7f4a2c0f5410 2023-12-21T03:44:01.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.601+0000 7f4a32328700 1 -- 172.21.15.148:0/2422091116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c059100 con 0x7f4a2c0f5410 2023-12-21T03:44:01.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.601+0000 7f4a28ff9700 1 --2- 172.21.15.148:0/2422091116 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4a1802ff60 0x7f4a18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:01.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.601+0000 7f4a28ff9700 1 -- 172.21.15.148:0/2422091116 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a1c041000 con 0x7f4a2c0f5410 2023-12-21T03:44:01.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.601+0000 7f4a2b7fe700 1 --2- 172.21.15.148:0/2422091116 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4a1802ff60 0x7f4a18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:01.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.602+0000 7f4a2b7fe700 1 --2- 172.21.15.148:0/2422091116 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4a1802ff60 0x7f4a18032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f4a140096a0 tx=0x7f4a14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:01.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.606+0000 7f4a28ff9700 1 -- 172.21.15.148:0/2422091116 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a1c014420 con 0x7f4a2c0f5410 2023-12-21T03:44:01.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.811+0000 7f4a32328700 1 -- 172.21.15.148:0/2422091116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a2c045ad0 con 0x7f4a2c0f5410 2023-12-21T03:44:01.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.812+0000 7f4a28ff9700 1 -- 172.21.15.148:0/2422091116 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a1c010230 con 0x7f4a2c0f5410 2023-12-21T03:44:01.813 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:01.813 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.813+0000 7f4a32328700 1 -- 172.21.15.148:0/2422091116 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4a1802ff60 msgr2=0x7f4a18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.813+0000 7f4a32328700 1 --2- 172.21.15.148:0/2422091116 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4a1802ff60 0x7f4a18032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f4a140096a0 tx=0x7f4a14006b40).stop 2023-12-21T03:44:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.813+0000 7f4a32328700 1 -- 172.21.15.148:0/2422091116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2c0f5410 msgr2=0x7f4a2c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.813+0000 7f4a32328700 1 --2- 172.21.15.148:0/2422091116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2c0f5410 0x7f4a2c109450 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f4a1c013510 tx=0x7f4a1c00b040).stop 2023-12-21T03:44:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.813+0000 7f4a32328700 1 -- 172.21.15.148:0/2422091116 shutdown_connections 2023-12-21T03:44:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.813+0000 7f4a32328700 1 --2- 172.21.15.148:0/2422091116 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4a1802ff60 0x7f4a18032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.813+0000 7f4a32328700 1 --2- 172.21.15.148:0/2422091116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2c0f5410 0x7f4a2c109450 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.814+0000 7f4a32328700 1 -- 172.21.15.148:0/2422091116 >> 172.21.15.148:0/2422091116 conn(0x7f4a2c0f0b70 msgr2=0x7f4a2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.814+0000 7f4a32328700 1 -- 172.21.15.148:0/2422091116 shutdown_connections 2023-12-21T03:44:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:01.814+0000 7f4a32328700 1 -- 172.21.15.148:0/2422091116 wait complete. 2023-12-21T03:44:01.816 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:00.443554+0000 mgr.smithi078.zidbgq (mgr.14162) 142 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:02.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:01.812780+0000 mon.smithi078 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.148:0/2422091116' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:03.224 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:03.225 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:02.443838+0000 mgr.smithi078.zidbgq (mgr.14162) 143 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.368+0000 7ff51f498700 1 -- 172.21.15.148:0/2944939851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff518060220 msgr2=0x7ff5180f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.368+0000 7ff51f498700 1 --2- 172.21.15.148:0/2944939851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff518060220 0x7ff5180f9150 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7ff508009fb0 tx=0x7ff508009b30).stop 2023-12-21T03:44:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.368+0000 7ff51f498700 1 -- 172.21.15.148:0/2944939851 shutdown_connections 2023-12-21T03:44:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.368+0000 7ff51f498700 1 --2- 172.21.15.148:0/2944939851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff518060220 0x7ff5180f9150 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.368+0000 7ff51f498700 1 -- 172.21.15.148:0/2944939851 >> 172.21.15.148:0/2944939851 conn(0x7ff5180f0b70 msgr2=0x7ff5180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.368+0000 7ff51f498700 1 -- 172.21.15.148:0/2944939851 shutdown_connections 2023-12-21T03:44:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.368+0000 7ff51f498700 1 -- 172.21.15.148:0/2944939851 wait complete. 2023-12-21T03:44:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.368+0000 7ff51f498700 1 Processor -- start 2023-12-21T03:44:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.369+0000 7ff51f498700 1 -- start start 2023-12-21T03:44:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.369+0000 7ff51f498700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff518060220 0x7ff5181066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.369+0000 7ff51f498700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff518106b90 con 0x7ff518060220 2023-12-21T03:44:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.369+0000 7ff51d234700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff518060220 0x7ff5181066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.369+0000 7ff51d234700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff518060220 0x7ff5181066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:50078/0 (socket says 172.21.15.148:50078) 2023-12-21T03:44:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.369+0000 7ff51d234700 1 -- 172.21.15.148:0/4160592498 learned_addr learned my addr 172.21.15.148:0/4160592498 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.369+0000 7ff51d234700 1 -- 172.21.15.148:0/4160592498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff508005040 con 0x7ff518060220 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.369+0000 7ff51d234700 1 --2- 172.21.15.148:0/4160592498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff518060220 0x7ff5181066a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7ff508013510 tx=0x7ff50800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.369+0000 7ff50dffb700 1 -- 172.21.15.148:0/4160592498 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff508009320 con 0x7ff518060220 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.370+0000 7ff51f498700 1 -- 172.21.15.148:0/4160592498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff518106d90 con 0x7ff518060220 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.370+0000 7ff50dffb700 1 -- 172.21.15.148:0/4160592498 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff508013f00 con 0x7ff518060220 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.370+0000 7ff50dffb700 1 -- 172.21.15.148:0/4160592498 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff508014dc0 con 0x7ff518060220 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.370+0000 7ff51f498700 1 -- 172.21.15.148:0/4160592498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff518107220 con 0x7ff518060220 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.370+0000 7ff50dffb700 1 -- 172.21.15.148:0/4160592498 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff508014420 con 0x7ff518060220 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.371+0000 7ff51f498700 1 -- 172.21.15.148:0/4160592498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff518045ad0 con 0x7ff518060220 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.371+0000 7ff50dffb700 1 --2- 172.21.15.148:0/4160592498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff50402ff60 0x7ff504032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.371+0000 7ff50dffb700 1 -- 172.21.15.148:0/4160592498 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff508041030 con 0x7ff518060220 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.371+0000 7ff51ca33700 1 --2- 172.21.15.148:0/4160592498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff50402ff60 0x7ff504032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.371+0000 7ff51ca33700 1 --2- 172.21.15.148:0/4160592498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff50402ff60 0x7ff504032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7ff5140096a0 tx=0x7ff514006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.375+0000 7ff50dffb700 1 -- 172.21.15.148:0/4160592498 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5080095f0 con 0x7ff518060220 2023-12-21T03:44:04.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.581+0000 7ff51f498700 1 -- 172.21.15.148:0/4160592498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5181080d0 con 0x7ff518060220 2023-12-21T03:44:04.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.581+0000 7ff50dffb700 1 -- 172.21.15.148:0/4160592498 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff50801cb10 con 0x7ff518060220 2023-12-21T03:44:04.583 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:04.583 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:04.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.583+0000 7ff51f498700 1 -- 172.21.15.148:0/4160592498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff50402ff60 msgr2=0x7ff504032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:04.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.583+0000 7ff51f498700 1 --2- 172.21.15.148:0/4160592498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff50402ff60 0x7ff504032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7ff5140096a0 tx=0x7ff514006b40).stop 2023-12-21T03:44:04.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.583+0000 7ff51f498700 1 -- 172.21.15.148:0/4160592498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff518060220 msgr2=0x7ff5181066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:04.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.583+0000 7ff51f498700 1 --2- 172.21.15.148:0/4160592498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff518060220 0x7ff5181066a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7ff508013510 tx=0x7ff50800b040).stop 2023-12-21T03:44:04.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.583+0000 7ff51f498700 1 -- 172.21.15.148:0/4160592498 shutdown_connections 2023-12-21T03:44:04.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.583+0000 7ff51f498700 1 --2- 172.21.15.148:0/4160592498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff50402ff60 0x7ff504032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:04.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.583+0000 7ff51f498700 1 --2- 172.21.15.148:0/4160592498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff518060220 0x7ff5181066a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:04.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.583+0000 7ff51f498700 1 -- 172.21.15.148:0/4160592498 >> 172.21.15.148:0/4160592498 conn(0x7ff5180f0b70 msgr2=0x7ff5180f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:04.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.583+0000 7ff51f498700 1 -- 172.21.15.148:0/4160592498 shutdown_connections 2023-12-21T03:44:04.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:04.583+0000 7ff51f498700 1 -- 172.21.15.148:0/4160592498 wait complete. 2023-12-21T03:44:04.585 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:05.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:04.444137+0000 mgr.smithi078.zidbgq (mgr.14162) 144 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:05.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:04.581960+0000 mon.smithi078 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.148:0/4160592498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:05.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:05.960 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:07.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.115+0000 7f5af0493700 1 -- 172.21.15.148:0/4263273337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae8060280 msgr2=0x7f5ae80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:07.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.115+0000 7f5af0493700 1 --2- 172.21.15.148:0/4263273337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae8060280 0x7f5ae80f90f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f5ae4009fb0 tx=0x7f5ae4009b30).stop 2023-12-21T03:44:07.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.115+0000 7f5af0493700 1 -- 172.21.15.148:0/4263273337 shutdown_connections 2023-12-21T03:44:07.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.115+0000 7f5af0493700 1 --2- 172.21.15.148:0/4263273337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae8060280 0x7f5ae80f90f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:07.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.115+0000 7f5af0493700 1 -- 172.21.15.148:0/4263273337 >> 172.21.15.148:0/4263273337 conn(0x7f5ae80f0b70 msgr2=0x7f5ae80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:07.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.115+0000 7f5af0493700 1 -- 172.21.15.148:0/4263273337 shutdown_connections 2023-12-21T03:44:07.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.115+0000 7f5af0493700 1 -- 172.21.15.148:0/4263273337 wait complete. 2023-12-21T03:44:07.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.116+0000 7f5af0493700 1 Processor -- start 2023-12-21T03:44:07.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.116+0000 7f5af0493700 1 -- start start 2023-12-21T03:44:07.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.116+0000 7f5af0493700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:07.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.116+0000 7f5af0493700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae8106bd0 con 0x7f5ae8060280 2023-12-21T03:44:07.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.116+0000 7f5aee22f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:07.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.116+0000 7f5aee22f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:50090/0 (socket says 172.21.15.148:50090) 2023-12-21T03:44:07.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.116+0000 7f5aee22f700 1 -- 172.21.15.148:0/830986379 learned_addr learned my addr 172.21.15.148:0/830986379 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:07.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.116+0000 7f5aee22f700 1 -- 172.21.15.148:0/830986379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4005040 con 0x7f5ae8060280 2023-12-21T03:44:07.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.116+0000 7f5aee22f700 1 --2- 172.21.15.148:0/830986379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f5ae4013510 tx=0x7f5ae400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:07.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.117+0000 7f5adeffd700 1 -- 172.21.15.148:0/830986379 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ae4009320 con 0x7f5ae8060280 2023-12-21T03:44:07.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.117+0000 7f5af0493700 1 -- 172.21.15.148:0/830986379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae8106dd0 con 0x7f5ae8060280 2023-12-21T03:44:07.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.117+0000 7f5adeffd700 1 -- 172.21.15.148:0/830986379 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ae4013f00 con 0x7f5ae8060280 2023-12-21T03:44:07.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.117+0000 7f5adeffd700 1 -- 172.21.15.148:0/830986379 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ae4014dc0 con 0x7f5ae8060280 2023-12-21T03:44:07.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.117+0000 7f5af0493700 1 -- 172.21.15.148:0/830986379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae8107260 con 0x7f5ae8060280 2023-12-21T03:44:07.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.117+0000 7f5adeffd700 1 -- 172.21.15.148:0/830986379 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ae401d3d0 con 0x7f5ae8060280 2023-12-21T03:44:07.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.117+0000 7f5af0493700 1 -- 172.21.15.148:0/830986379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae8045ad0 con 0x7f5ae8060280 2023-12-21T03:44:07.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.118+0000 7f5adeffd700 1 --2- 172.21.15.148:0/830986379 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ad402ff60 0x7f5ad4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:07.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.118+0000 7f5adeffd700 1 -- 172.21.15.148:0/830986379 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ae40413c0 con 0x7f5ae8060280 2023-12-21T03:44:07.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.118+0000 7f5aeda2e700 1 --2- 172.21.15.148:0/830986379 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ad402ff60 0x7f5ad4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:07.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.118+0000 7f5aeda2e700 1 --2- 172.21.15.148:0/830986379 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ad402ff60 0x7f5ad4032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f5ad80096a0 tx=0x7f5ad8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:07.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.122+0000 7f5adeffd700 1 -- 172.21.15.148:0/830986379 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ae401cd50 con 0x7f5ae8060280 2023-12-21T03:44:07.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.329+0000 7f5af0493700 1 -- 172.21.15.148:0/830986379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ae8108110 con 0x7f5ae8060280 2023-12-21T03:44:07.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.330+0000 7f5adeffd700 1 -- 172.21.15.148:0/830986379 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ae401cb70 con 0x7f5ae8060280 2023-12-21T03:44:07.331 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:07.331 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:07.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.331+0000 7f5af0493700 1 -- 172.21.15.148:0/830986379 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ad402ff60 msgr2=0x7f5ad4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:07.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.331+0000 7f5af0493700 1 --2- 172.21.15.148:0/830986379 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ad402ff60 0x7f5ad4032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f5ad80096a0 tx=0x7f5ad8006b40).stop 2023-12-21T03:44:07.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.331+0000 7f5af0493700 1 -- 172.21.15.148:0/830986379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae8060280 msgr2=0x7f5ae81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:07.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.331+0000 7f5af0493700 1 --2- 172.21.15.148:0/830986379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f5ae4013510 tx=0x7f5ae400b040).stop 2023-12-21T03:44:07.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.331+0000 7f5af0493700 1 -- 172.21.15.148:0/830986379 shutdown_connections 2023-12-21T03:44:07.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.331+0000 7f5af0493700 1 --2- 172.21.15.148:0/830986379 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ad402ff60 0x7f5ad4032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:07.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.331+0000 7f5af0493700 1 --2- 172.21.15.148:0/830986379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:07.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.331+0000 7f5af0493700 1 -- 172.21.15.148:0/830986379 >> 172.21.15.148:0/830986379 conn(0x7f5ae80f0b70 msgr2=0x7f5ae80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:07.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.331+0000 7f5af0493700 1 -- 172.21.15.148:0/830986379 shutdown_connections 2023-12-21T03:44:07.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:07.331+0000 7f5af0493700 1 -- 172.21.15.148:0/830986379 wait complete. 2023-12-21T03:44:07.333 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:06.444462+0000 mgr.smithi078.zidbgq (mgr.14162) 145 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:07.330544+0000 mon.smithi078 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.148:0/830986379' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:08.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:08.725 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:08.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:08.444790+0000 mgr.smithi078.zidbgq (mgr.14162) 146 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:09.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.859+0000 7fc742887700 1 -- 172.21.15.148:0/3243698155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc73c060280 msgr2=0x7fc73c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:09.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.859+0000 7fc742887700 1 --2- 172.21.15.148:0/3243698155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc73c060280 0x7fc73c0f90f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fc730009fb0 tx=0x7fc730009b30).stop 2023-12-21T03:44:09.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.859+0000 7fc742887700 1 -- 172.21.15.148:0/3243698155 shutdown_connections 2023-12-21T03:44:09.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.859+0000 7fc742887700 1 --2- 172.21.15.148:0/3243698155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc73c060280 0x7fc73c0f90f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:09.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.859+0000 7fc742887700 1 -- 172.21.15.148:0/3243698155 >> 172.21.15.148:0/3243698155 conn(0x7fc73c0f0b70 msgr2=0x7fc73c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.859+0000 7fc742887700 1 -- 172.21.15.148:0/3243698155 shutdown_connections 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.859+0000 7fc742887700 1 -- 172.21.15.148:0/3243698155 wait complete. 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.860+0000 7fc742887700 1 Processor -- start 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.860+0000 7fc742887700 1 -- start start 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.860+0000 7fc742887700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc73c060280 0x7fc73c1066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.860+0000 7fc742887700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c106bb0 con 0x7fc73c060280 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.860+0000 7fc73bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc73c060280 0x7fc73c1066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.861+0000 7fc73bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc73c060280 0x7fc73c1066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46820/0 (socket says 172.21.15.148:46820) 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.861+0000 7fc73bfff700 1 -- 172.21.15.148:0/341422209 learned_addr learned my addr 172.21.15.148:0/341422209 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.861+0000 7fc73bfff700 1 -- 172.21.15.148:0/341422209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc730005040 con 0x7fc73c060280 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.861+0000 7fc73bfff700 1 --2- 172.21.15.148:0/341422209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc73c060280 0x7fc73c1066c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fc730009830 tx=0x7fc73000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.861+0000 7fc738ff9700 1 -- 172.21.15.148:0/341422209 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc730009320 con 0x7fc73c060280 2023-12-21T03:44:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.861+0000 7fc742887700 1 -- 172.21.15.148:0/341422209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c106db0 con 0x7fc73c060280 2023-12-21T03:44:09.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.861+0000 7fc738ff9700 1 -- 172.21.15.148:0/341422209 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7300136c0 con 0x7fc73c060280 2023-12-21T03:44:09.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.861+0000 7fc738ff9700 1 -- 172.21.15.148:0/341422209 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc73001ee90 con 0x7fc73c060280 2023-12-21T03:44:09.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.861+0000 7fc742887700 1 -- 172.21.15.148:0/341422209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c107240 con 0x7fc73c060280 2023-12-21T03:44:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.862+0000 7fc738ff9700 1 -- 172.21.15.148:0/341422209 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc73001e420 con 0x7fc73c060280 2023-12-21T03:44:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.862+0000 7fc742887700 1 -- 172.21.15.148:0/341422209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c045ad0 con 0x7fc73c060280 2023-12-21T03:44:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.862+0000 7fc738ff9700 1 --2- 172.21.15.148:0/341422209 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc72402ff10 0x7fc7240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.862+0000 7fc738ff9700 1 -- 172.21.15.148:0/341422209 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7300410b0 con 0x7fc73c060280 2023-12-21T03:44:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.862+0000 7fc73b7fe700 1 --2- 172.21.15.148:0/341422209 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc72402ff10 0x7fc7240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.863+0000 7fc73b7fe700 1 --2- 172.21.15.148:0/341422209 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc72402ff10 0x7fc7240323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc72c0096a0 tx=0x7fc72c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:09.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:09.867+0000 7fc738ff9700 1 -- 172.21.15.148:0/341422209 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7300095f0 con 0x7fc73c060280 2023-12-21T03:44:10.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.073+0000 7fc742887700 1 -- 172.21.15.148:0/341422209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc73c108090 con 0x7fc73c060280 2023-12-21T03:44:10.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.074+0000 7fc738ff9700 1 -- 172.21.15.148:0/341422209 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc73002c2d0 con 0x7fc73c060280 2023-12-21T03:44:10.075 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:10.075 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:10.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.075+0000 7fc742887700 1 -- 172.21.15.148:0/341422209 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc72402ff10 msgr2=0x7fc7240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:10.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.075+0000 7fc742887700 1 --2- 172.21.15.148:0/341422209 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc72402ff10 0x7fc7240323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc72c0096a0 tx=0x7fc72c006b40).stop 2023-12-21T03:44:10.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.075+0000 7fc742887700 1 -- 172.21.15.148:0/341422209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc73c060280 msgr2=0x7fc73c1066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:10.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.075+0000 7fc742887700 1 --2- 172.21.15.148:0/341422209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc73c060280 0x7fc73c1066c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fc730009830 tx=0x7fc73000b040).stop 2023-12-21T03:44:10.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.075+0000 7fc742887700 1 -- 172.21.15.148:0/341422209 shutdown_connections 2023-12-21T03:44:10.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.075+0000 7fc742887700 1 --2- 172.21.15.148:0/341422209 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc72402ff10 0x7fc7240323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:10.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.075+0000 7fc742887700 1 --2- 172.21.15.148:0/341422209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc73c060280 0x7fc73c1066c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:10.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.075+0000 7fc742887700 1 -- 172.21.15.148:0/341422209 >> 172.21.15.148:0/341422209 conn(0x7fc73c0f0b70 msgr2=0x7fc73c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:10.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.076+0000 7fc742887700 1 -- 172.21.15.148:0/341422209 shutdown_connections 2023-12-21T03:44:10.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:10.076+0000 7fc742887700 1 -- 172.21.15.148:0/341422209 wait complete. 2023-12-21T03:44:10.077 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:10.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:10 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:10.074662+0000 mon.smithi078 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.148:0/341422209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:11.455 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:11.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:11.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:10.445122+0000 mgr.smithi078.zidbgq (mgr.14162) 147 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:12.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.606+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2307602464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940f5410 msgr2=0x7f6a940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:12.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.606+0000 7f6a9ac1a700 1 --2- 172.21.15.148:0/2307602464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940f5410 0x7f6a940f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f6a88009fb0 tx=0x7f6a88009b30).stop 2023-12-21T03:44:12.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.606+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2307602464 shutdown_connections 2023-12-21T03:44:12.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.606+0000 7f6a9ac1a700 1 --2- 172.21.15.148:0/2307602464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940f5410 0x7f6a940f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:12.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.606+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2307602464 >> 172.21.15.148:0/2307602464 conn(0x7f6a940f0b70 msgr2=0x7f6a940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:12.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.606+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2307602464 shutdown_connections 2023-12-21T03:44:12.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.606+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2307602464 wait complete. 2023-12-21T03:44:12.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.607+0000 7f6a9ac1a700 1 Processor -- start 2023-12-21T03:44:12.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.607+0000 7f6a9ac1a700 1 -- start start 2023-12-21T03:44:12.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.607+0000 7f6a9ac1a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940f5410 0x7f6a94109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:12.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.607+0000 7f6a9ac1a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a94060220 con 0x7f6a940f5410 2023-12-21T03:44:12.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.608+0000 7f6a989b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940f5410 0x7f6a94109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:12.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.608+0000 7f6a989b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940f5410 0x7f6a94109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46830/0 (socket says 172.21.15.148:46830) 2023-12-21T03:44:12.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.608+0000 7f6a989b6700 1 -- 172.21.15.148:0/2189435404 learned_addr learned my addr 172.21.15.148:0/2189435404 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:12.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.608+0000 7f6a989b6700 1 -- 172.21.15.148:0/2189435404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a88005040 con 0x7f6a940f5410 2023-12-21T03:44:12.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.608+0000 7f6a989b6700 1 --2- 172.21.15.148:0/2189435404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940f5410 0x7f6a94109450 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f6a88013510 tx=0x7f6a8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:12.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.608+0000 7f6a917fa700 1 -- 172.21.15.148:0/2189435404 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a88009320 con 0x7f6a940f5410 2023-12-21T03:44:12.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.608+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2189435404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a94060420 con 0x7f6a940f5410 2023-12-21T03:44:12.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.608+0000 7f6a917fa700 1 -- 172.21.15.148:0/2189435404 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a88013f00 con 0x7f6a940f5410 2023-12-21T03:44:12.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.608+0000 7f6a917fa700 1 -- 172.21.15.148:0/2189435404 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a88014d90 con 0x7f6a940f5410 2023-12-21T03:44:12.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.608+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2189435404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a94109b00 con 0x7f6a940f5410 2023-12-21T03:44:12.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.609+0000 7f6a917fa700 1 -- 172.21.15.148:0/2189435404 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a880095f0 con 0x7f6a940f5410 2023-12-21T03:44:12.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.609+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2189435404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a94059100 con 0x7f6a940f5410 2023-12-21T03:44:12.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.609+0000 7f6a917fa700 1 --2- 172.21.15.148:0/2189435404 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f6a8402ffb0 0x7f6a84032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:12.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.609+0000 7f6a917fa700 1 -- 172.21.15.148:0/2189435404 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a880410c0 con 0x7f6a940f5410 2023-12-21T03:44:12.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.610+0000 7f6a93fff700 1 --2- 172.21.15.148:0/2189435404 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f6a8402ffb0 0x7f6a84032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:12.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.610+0000 7f6a93fff700 1 --2- 172.21.15.148:0/2189435404 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f6a8402ffb0 0x7f6a84032470 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f6a800096a0 tx=0x7f6a80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:12.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.614+0000 7f6a917fa700 1 -- 172.21.15.148:0/2189435404 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a88014420 con 0x7f6a940f5410 2023-12-21T03:44:12.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:12.445368+0000 mgr.smithi078.zidbgq (mgr.14162) 148 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:12.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.821+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2189435404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a94045ad0 con 0x7f6a940f5410 2023-12-21T03:44:12.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.822+0000 7f6a917fa700 1 -- 172.21.15.148:0/2189435404 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6a88010230 con 0x7f6a940f5410 2023-12-21T03:44:12.823 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:12.823 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:12.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.823+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2189435404 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f6a8402ffb0 msgr2=0x7f6a84032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:12.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.823+0000 7f6a9ac1a700 1 --2- 172.21.15.148:0/2189435404 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f6a8402ffb0 0x7f6a84032470 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f6a800096a0 tx=0x7f6a80006b40).stop 2023-12-21T03:44:12.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.823+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2189435404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940f5410 msgr2=0x7f6a94109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:12.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.823+0000 7f6a9ac1a700 1 --2- 172.21.15.148:0/2189435404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940f5410 0x7f6a94109450 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f6a88013510 tx=0x7f6a8800b040).stop 2023-12-21T03:44:12.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.823+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2189435404 shutdown_connections 2023-12-21T03:44:12.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.823+0000 7f6a9ac1a700 1 --2- 172.21.15.148:0/2189435404 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f6a8402ffb0 0x7f6a84032470 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:12.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.823+0000 7f6a9ac1a700 1 --2- 172.21.15.148:0/2189435404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940f5410 0x7f6a94109450 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:12.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.823+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2189435404 >> 172.21.15.148:0/2189435404 conn(0x7f6a940f0b70 msgr2=0x7f6a940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:12.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.823+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2189435404 shutdown_connections 2023-12-21T03:44:12.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:12.823+0000 7f6a9ac1a700 1 -- 172.21.15.148:0/2189435404 wait complete. 2023-12-21T03:44:12.825 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:12.822660+0000 mon.smithi078 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.148:0/2189435404' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:14.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:14.233 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:15.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.372+0000 7f2b9a776700 1 -- 172.21.15.148:0/3021520934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b940f6510 msgr2=0x7f2b940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:15.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.372+0000 7f2b9a776700 1 --2- 172.21.15.148:0/3021520934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b940f6510 0x7f2b940f6930 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f2b84009fb0 tx=0x7f2b84009b30).stop 2023-12-21T03:44:15.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.372+0000 7f2b9a776700 1 -- 172.21.15.148:0/3021520934 shutdown_connections 2023-12-21T03:44:15.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.372+0000 7f2b9a776700 1 --2- 172.21.15.148:0/3021520934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b940f6510 0x7f2b940f6930 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:15.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.372+0000 7f2b9a776700 1 -- 172.21.15.148:0/3021520934 >> 172.21.15.148:0/3021520934 conn(0x7f2b940f1c70 msgr2=0x7f2b940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:15.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.372+0000 7f2b9a776700 1 -- 172.21.15.148:0/3021520934 shutdown_connections 2023-12-21T03:44:15.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.372+0000 7f2b9a776700 1 -- 172.21.15.148:0/3021520934 wait complete. 2023-12-21T03:44:15.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.373+0000 7f2b9a776700 1 Processor -- start 2023-12-21T03:44:15.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.373+0000 7f2b9a776700 1 -- start start 2023-12-21T03:44:15.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.373+0000 7f2b9a776700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b940f6510 0x7f2b94106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:15.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.373+0000 7f2b9a776700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b94106c20 con 0x7f2b940f6510 2023-12-21T03:44:15.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.373+0000 7f2b93fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b940f6510 0x7f2b94106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:15.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.373+0000 7f2b93fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b940f6510 0x7f2b94106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46856/0 (socket says 172.21.15.148:46856) 2023-12-21T03:44:15.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.373+0000 7f2b93fff700 1 -- 172.21.15.148:0/688216532 learned_addr learned my addr 172.21.15.148:0/688216532 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:15.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.374+0000 7f2b93fff700 1 -- 172.21.15.148:0/688216532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b84005040 con 0x7f2b940f6510 2023-12-21T03:44:15.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.375+0000 7f2b93fff700 1 --2- 172.21.15.148:0/688216532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b940f6510 0x7f2b94106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f2b84013510 tx=0x7f2b8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:15.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.375+0000 7f2b90ff9700 1 -- 172.21.15.148:0/688216532 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b84009320 con 0x7f2b940f6510 2023-12-21T03:44:15.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.375+0000 7f2b9a776700 1 -- 172.21.15.148:0/688216532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b94106e20 con 0x7f2b940f6510 2023-12-21T03:44:15.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.375+0000 7f2b90ff9700 1 -- 172.21.15.148:0/688216532 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b84013f00 con 0x7f2b940f6510 2023-12-21T03:44:15.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.375+0000 7f2b90ff9700 1 -- 172.21.15.148:0/688216532 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b84014d90 con 0x7f2b940f6510 2023-12-21T03:44:15.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.375+0000 7f2b9a776700 1 -- 172.21.15.148:0/688216532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b941072b0 con 0x7f2b940f6510 2023-12-21T03:44:15.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.376+0000 7f2b9a776700 1 -- 172.21.15.148:0/688216532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b94045ad0 con 0x7f2b940f6510 2023-12-21T03:44:15.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.377+0000 7f2b90ff9700 1 -- 172.21.15.148:0/688216532 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b840095f0 con 0x7f2b940f6510 2023-12-21T03:44:15.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.377+0000 7f2b90ff9700 1 --2- 172.21.15.148:0/688216532 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2b7c02ff60 0x7f2b7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:15.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.377+0000 7f2b90ff9700 1 -- 172.21.15.148:0/688216532 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b840418d0 con 0x7f2b940f6510 2023-12-21T03:44:15.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.378+0000 7f2b937fe700 1 --2- 172.21.15.148:0/688216532 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2b7c02ff60 0x7f2b7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:15.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.378+0000 7f2b937fe700 1 --2- 172.21.15.148:0/688216532 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2b7c02ff60 0x7f2b7c032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f2b880096a0 tx=0x7f2b88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:15.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.381+0000 7f2b90ff9700 1 -- 172.21.15.148:0/688216532 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b8401c790 con 0x7f2b940f6510 2023-12-21T03:44:15.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:14.445699+0000 mgr.smithi078.zidbgq (mgr.14162) 149 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:15.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.586+0000 7f2b9a776700 1 -- 172.21.15.148:0/688216532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b940fc050 con 0x7f2b940f6510 2023-12-21T03:44:15.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.586+0000 7f2b90ff9700 1 -- 172.21.15.148:0/688216532 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b8401c790 con 0x7f2b940f6510 2023-12-21T03:44:15.588 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:15.588 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.588+0000 7f2b9a776700 1 -- 172.21.15.148:0/688216532 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2b7c02ff60 msgr2=0x7f2b7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.588+0000 7f2b9a776700 1 --2- 172.21.15.148:0/688216532 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2b7c02ff60 0x7f2b7c032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f2b880096a0 tx=0x7f2b88006b40).stop 2023-12-21T03:44:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.588+0000 7f2b9a776700 1 -- 172.21.15.148:0/688216532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b940f6510 msgr2=0x7f2b94106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.588+0000 7f2b9a776700 1 --2- 172.21.15.148:0/688216532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b940f6510 0x7f2b94106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f2b84013510 tx=0x7f2b8400b040).stop 2023-12-21T03:44:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.588+0000 7f2b9a776700 1 -- 172.21.15.148:0/688216532 shutdown_connections 2023-12-21T03:44:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.588+0000 7f2b9a776700 1 --2- 172.21.15.148:0/688216532 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f2b7c02ff60 0x7f2b7c032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.588+0000 7f2b9a776700 1 --2- 172.21.15.148:0/688216532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b940f6510 0x7f2b94106730 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.588+0000 7f2b9a776700 1 -- 172.21.15.148:0/688216532 >> 172.21.15.148:0/688216532 conn(0x7f2b940f1c70 msgr2=0x7f2b940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:15.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.588+0000 7f2b9a776700 1 -- 172.21.15.148:0/688216532 shutdown_connections 2023-12-21T03:44:15.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:15.588+0000 7f2b9a776700 1 -- 172.21.15.148:0/688216532 wait complete. 2023-12-21T03:44:15.590 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:16.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:16 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:15.587262+0000 mon.smithi078 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.148:0/688216532' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:16.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:16.953 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:17.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:16.446061+0000 mgr.smithi078.zidbgq (mgr.14162) 150 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:18.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.143+0000 7f5cc2712700 1 -- 172.21.15.148:0/505349913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cbc0f6510 msgr2=0x7f5cbc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:18.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.143+0000 7f5cc2712700 1 --2- 172.21.15.148:0/505349913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cbc0f6510 0x7f5cbc0f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f5cb0009fb0 tx=0x7f5cb0009b30).stop 2023-12-21T03:44:18.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.143+0000 7f5cc2712700 1 -- 172.21.15.148:0/505349913 shutdown_connections 2023-12-21T03:44:18.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.143+0000 7f5cc2712700 1 --2- 172.21.15.148:0/505349913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cbc0f6510 0x7f5cbc0f6930 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:18.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.143+0000 7f5cc2712700 1 -- 172.21.15.148:0/505349913 >> 172.21.15.148:0/505349913 conn(0x7f5cbc0f1c70 msgr2=0x7f5cbc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:18.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.143+0000 7f5cc2712700 1 -- 172.21.15.148:0/505349913 shutdown_connections 2023-12-21T03:44:18.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.144+0000 7f5cc2712700 1 -- 172.21.15.148:0/505349913 wait complete. 2023-12-21T03:44:18.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.144+0000 7f5cc2712700 1 Processor -- start 2023-12-21T03:44:18.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.144+0000 7f5cc2712700 1 -- start start 2023-12-21T03:44:18.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.144+0000 7f5cc2712700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cbc0f6510 0x7f5cbc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:18.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.144+0000 7f5cc2712700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc106c20 con 0x7f5cbc0f6510 2023-12-21T03:44:18.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.145+0000 7f5cbbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cbc0f6510 0x7f5cbc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:18.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.145+0000 7f5cbbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cbc0f6510 0x7f5cbc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:46862/0 (socket says 172.21.15.148:46862) 2023-12-21T03:44:18.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.145+0000 7f5cbbfff700 1 -- 172.21.15.148:0/2743504397 learned_addr learned my addr 172.21.15.148:0/2743504397 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:18.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.145+0000 7f5cbbfff700 1 -- 172.21.15.148:0/2743504397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb0005040 con 0x7f5cbc0f6510 2023-12-21T03:44:18.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.145+0000 7f5cbbfff700 1 --2- 172.21.15.148:0/2743504397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cbc0f6510 0x7f5cbc106730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f5cb0013510 tx=0x7f5cb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:18.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.145+0000 7f5cb8ff9700 1 -- 172.21.15.148:0/2743504397 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cb0009320 con 0x7f5cbc0f6510 2023-12-21T03:44:18.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.145+0000 7f5cc2712700 1 -- 172.21.15.148:0/2743504397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cbc106e20 con 0x7f5cbc0f6510 2023-12-21T03:44:18.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.145+0000 7f5cb8ff9700 1 -- 172.21.15.148:0/2743504397 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cb0013f00 con 0x7f5cbc0f6510 2023-12-21T03:44:18.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.145+0000 7f5cb8ff9700 1 -- 172.21.15.148:0/2743504397 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cb0014d90 con 0x7f5cbc0f6510 2023-12-21T03:44:18.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.145+0000 7f5cc2712700 1 -- 172.21.15.148:0/2743504397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cbc1072b0 con 0x7f5cbc0f6510 2023-12-21T03:44:18.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.146+0000 7f5cc2712700 1 -- 172.21.15.148:0/2743504397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cbc045ad0 con 0x7f5cbc0f6510 2023-12-21T03:44:18.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.146+0000 7f5cb8ff9700 1 -- 172.21.15.148:0/2743504397 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cb00095f0 con 0x7f5cbc0f6510 2023-12-21T03:44:18.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.146+0000 7f5cb8ff9700 1 --2- 172.21.15.148:0/2743504397 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ca402ff60 0x7f5ca4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:18.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.146+0000 7f5cb8ff9700 1 -- 172.21.15.148:0/2743504397 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cb0041080 con 0x7f5cbc0f6510 2023-12-21T03:44:18.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.147+0000 7f5cbb7fe700 1 --2- 172.21.15.148:0/2743504397 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ca402ff60 0x7f5ca4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:18.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.147+0000 7f5cbb7fe700 1 --2- 172.21.15.148:0/2743504397 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ca402ff60 0x7f5ca4032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f5cac0096a0 tx=0x7f5cac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:18.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.151+0000 7f5cb8ff9700 1 -- 172.21.15.148:0/2743504397 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cb001d7f0 con 0x7f5cbc0f6510 2023-12-21T03:44:18.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.358+0000 7f5cc2712700 1 -- 172.21.15.148:0/2743504397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cbc0fbf60 con 0x7f5cbc0f6510 2023-12-21T03:44:18.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.359+0000 7f5cb8ff9700 1 -- 172.21.15.148:0/2743504397 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cb0010230 con 0x7f5cbc0f6510 2023-12-21T03:44:18.360 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:18.360 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:18.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.361+0000 7f5cc2712700 1 -- 172.21.15.148:0/2743504397 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ca402ff60 msgr2=0x7f5ca4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:18.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.361+0000 7f5cc2712700 1 --2- 172.21.15.148:0/2743504397 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ca402ff60 0x7f5ca4032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f5cac0096a0 tx=0x7f5cac006b40).stop 2023-12-21T03:44:18.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.361+0000 7f5cc2712700 1 -- 172.21.15.148:0/2743504397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cbc0f6510 msgr2=0x7f5cbc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:18.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.361+0000 7f5cc2712700 1 --2- 172.21.15.148:0/2743504397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cbc0f6510 0x7f5cbc106730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f5cb0013510 tx=0x7f5cb000b040).stop 2023-12-21T03:44:18.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.361+0000 7f5cc2712700 1 -- 172.21.15.148:0/2743504397 shutdown_connections 2023-12-21T03:44:18.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.361+0000 7f5cc2712700 1 --2- 172.21.15.148:0/2743504397 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5ca402ff60 0x7f5ca4032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:18.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.361+0000 7f5cc2712700 1 --2- 172.21.15.148:0/2743504397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cbc0f6510 0x7f5cbc106730 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:18.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.361+0000 7f5cc2712700 1 -- 172.21.15.148:0/2743504397 >> 172.21.15.148:0/2743504397 conn(0x7f5cbc0f1c70 msgr2=0x7f5cbc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:18.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.361+0000 7f5cc2712700 1 -- 172.21.15.148:0/2743504397 shutdown_connections 2023-12-21T03:44:18.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:18.361+0000 7f5cc2712700 1 -- 172.21.15.148:0/2743504397 wait complete. 2023-12-21T03:44:18.363 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:18.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:18 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:18.360141+0000 mon.smithi078 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.148:0/2743504397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:19.729 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:19.729 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:19.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:18.446400+0000 mgr.smithi078.zidbgq (mgr.14162) 151 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:20.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.846+0000 7fb9197f9700 1 -- 172.21.15.148:0/941690410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9140f5410 msgr2=0x7fb9140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:20.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.846+0000 7fb9197f9700 1 --2- 172.21.15.148:0/941690410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9140f5410 0x7fb9140f5830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fb904009fb0 tx=0x7fb904009b30).stop 2023-12-21T03:44:20.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.846+0000 7fb9197f9700 1 -- 172.21.15.148:0/941690410 shutdown_connections 2023-12-21T03:44:20.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.846+0000 7fb9197f9700 1 --2- 172.21.15.148:0/941690410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9140f5410 0x7fb9140f5830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:20.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.846+0000 7fb9197f9700 1 -- 172.21.15.148:0/941690410 >> 172.21.15.148:0/941690410 conn(0x7fb9140f0b70 msgr2=0x7fb9140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.846+0000 7fb9197f9700 1 -- 172.21.15.148:0/941690410 shutdown_connections 2023-12-21T03:44:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.847+0000 7fb9197f9700 1 -- 172.21.15.148:0/941690410 wait complete. 2023-12-21T03:44:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.847+0000 7fb9197f9700 1 Processor -- start 2023-12-21T03:44:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.847+0000 7fb9197f9700 1 -- start start 2023-12-21T03:44:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.847+0000 7fb9197f9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9140f5410 0x7fb914106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.847+0000 7fb9197f9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb914106c20 con 0x7fb9140f5410 2023-12-21T03:44:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.848+0000 7fb912ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9140f5410 0x7fb914106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.848+0000 7fb912ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9140f5410 0x7fb914106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:60892/0 (socket says 172.21.15.148:60892) 2023-12-21T03:44:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.848+0000 7fb912ffd700 1 -- 172.21.15.148:0/2362166992 learned_addr learned my addr 172.21.15.148:0/2362166992 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.848+0000 7fb912ffd700 1 -- 172.21.15.148:0/2362166992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb904005040 con 0x7fb9140f5410 2023-12-21T03:44:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.848+0000 7fb912ffd700 1 --2- 172.21.15.148:0/2362166992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9140f5410 0x7fb914106730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fb904013510 tx=0x7fb90400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.848+0000 7fb8fb7fe700 1 -- 172.21.15.148:0/2362166992 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb904009320 con 0x7fb9140f5410 2023-12-21T03:44:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.848+0000 7fb9197f9700 1 -- 172.21.15.148:0/2362166992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb914106e20 con 0x7fb9140f5410 2023-12-21T03:44:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.848+0000 7fb8fb7fe700 1 -- 172.21.15.148:0/2362166992 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb904013f00 con 0x7fb9140f5410 2023-12-21T03:44:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.848+0000 7fb8fb7fe700 1 -- 172.21.15.148:0/2362166992 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb904014d90 con 0x7fb9140f5410 2023-12-21T03:44:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.848+0000 7fb9197f9700 1 -- 172.21.15.148:0/2362166992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9141072b0 con 0x7fb9140f5410 2023-12-21T03:44:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.849+0000 7fb8fb7fe700 1 -- 172.21.15.148:0/2362166992 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9040095f0 con 0x7fb9140f5410 2023-12-21T03:44:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.849+0000 7fb9197f9700 1 -- 172.21.15.148:0/2362166992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb914045ad0 con 0x7fb9140f5410 2023-12-21T03:44:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.849+0000 7fb8fb7fe700 1 --2- 172.21.15.148:0/2362166992 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb8fc02ff60 0x7fb8fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.849+0000 7fb8fb7fe700 1 -- 172.21.15.148:0/2362166992 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb904040fc0 con 0x7fb9140f5410 2023-12-21T03:44:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.849+0000 7fb9127fc700 1 --2- 172.21.15.148:0/2362166992 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb8fc02ff60 0x7fb8fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.850+0000 7fb9127fc700 1 --2- 172.21.15.148:0/2362166992 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb8fc02ff60 0x7fb8fc032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fb9080096a0 tx=0x7fb908006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:20.854+0000 7fb8fb7fe700 1 -- 172.21.15.148:0/2362166992 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb904014420 con 0x7fb9140f5410 2023-12-21T03:44:21.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.061+0000 7fb9197f9700 1 -- 172.21.15.148:0/2362166992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9140fae60 con 0x7fb9140f5410 2023-12-21T03:44:21.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.062+0000 7fb8fb7fe700 1 -- 172.21.15.148:0/2362166992 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb904010230 con 0x7fb9140f5410 2023-12-21T03:44:21.063 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:21.063 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:21.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.063+0000 7fb9197f9700 1 -- 172.21.15.148:0/2362166992 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb8fc02ff60 msgr2=0x7fb8fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:21.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.063+0000 7fb9197f9700 1 --2- 172.21.15.148:0/2362166992 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb8fc02ff60 0x7fb8fc032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fb9080096a0 tx=0x7fb908006b40).stop 2023-12-21T03:44:21.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.063+0000 7fb9197f9700 1 -- 172.21.15.148:0/2362166992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9140f5410 msgr2=0x7fb914106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:21.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.063+0000 7fb9197f9700 1 --2- 172.21.15.148:0/2362166992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9140f5410 0x7fb914106730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fb904013510 tx=0x7fb90400b040).stop 2023-12-21T03:44:21.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.063+0000 7fb9197f9700 1 -- 172.21.15.148:0/2362166992 shutdown_connections 2023-12-21T03:44:21.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.063+0000 7fb9197f9700 1 --2- 172.21.15.148:0/2362166992 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb8fc02ff60 0x7fb8fc032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:21.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.063+0000 7fb9197f9700 1 --2- 172.21.15.148:0/2362166992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9140f5410 0x7fb914106730 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:21.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.063+0000 7fb9197f9700 1 -- 172.21.15.148:0/2362166992 >> 172.21.15.148:0/2362166992 conn(0x7fb9140f0b70 msgr2=0x7fb9140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:21.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.063+0000 7fb9197f9700 1 -- 172.21.15.148:0/2362166992 shutdown_connections 2023-12-21T03:44:21.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:21.063+0000 7fb9197f9700 1 -- 172.21.15.148:0/2362166992 wait complete. 2023-12-21T03:44:21.065 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:20.446630+0000 mgr.smithi078.zidbgq (mgr.14162) 152 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:21.062569+0000 mon.smithi078 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.148:0/2362166992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:22.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:22.449 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:22.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:22.446954+0000 mgr.smithi078.zidbgq (mgr.14162) 153 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.530+0000 7f09909ea700 1 -- 172.21.15.148:0/198941965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f098c0601e0 msgr2=0x7f098c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.530+0000 7f09909ea700 1 --2- 172.21.15.148:0/198941965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f098c0601e0 0x7f098c0f8f20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f097c009fb0 tx=0x7f097c009b30).stop 2023-12-21T03:44:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.530+0000 7f09909ea700 1 -- 172.21.15.148:0/198941965 shutdown_connections 2023-12-21T03:44:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.530+0000 7f09909ea700 1 --2- 172.21.15.148:0/198941965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f098c0601e0 0x7f098c0f8f20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.530+0000 7f09909ea700 1 -- 172.21.15.148:0/198941965 >> 172.21.15.148:0/198941965 conn(0x7f098c0f09a0 msgr2=0x7f098c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.530+0000 7f09909ea700 1 -- 172.21.15.148:0/198941965 shutdown_connections 2023-12-21T03:44:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.530+0000 7f09909ea700 1 -- 172.21.15.148:0/198941965 wait complete. 2023-12-21T03:44:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.530+0000 7f09909ea700 1 Processor -- start 2023-12-21T03:44:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.531+0000 7f09909ea700 1 -- start start 2023-12-21T03:44:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.531+0000 7f09909ea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f098c0601e0 0x7f098c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.531+0000 7f09909ea700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c1069f0 con 0x7f098c0601e0 2023-12-21T03:44:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.531+0000 7f098b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f098c0601e0 0x7f098c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.531+0000 7f098b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f098c0601e0 0x7f098c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:60902/0 (socket says 172.21.15.148:60902) 2023-12-21T03:44:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.531+0000 7f098b7fe700 1 -- 172.21.15.148:0/2484977137 learned_addr learned my addr 172.21.15.148:0/2484977137 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.531+0000 7f098b7fe700 1 -- 172.21.15.148:0/2484977137 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c005040 con 0x7f098c0601e0 2023-12-21T03:44:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.531+0000 7f098b7fe700 1 --2- 172.21.15.148:0/2484977137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f098c0601e0 0x7f098c106500 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f097c013510 tx=0x7f097c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.532+0000 7f0973fff700 1 -- 172.21.15.148:0/2484977137 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f097c009320 con 0x7f098c0601e0 2023-12-21T03:44:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.532+0000 7f09909ea700 1 -- 172.21.15.148:0/2484977137 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098c106bf0 con 0x7f098c0601e0 2023-12-21T03:44:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.532+0000 7f0973fff700 1 -- 172.21.15.148:0/2484977137 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f097c013f00 con 0x7f098c0601e0 2023-12-21T03:44:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.532+0000 7f0973fff700 1 -- 172.21.15.148:0/2484977137 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f097c014d90 con 0x7f098c0601e0 2023-12-21T03:44:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.532+0000 7f09909ea700 1 -- 172.21.15.148:0/2484977137 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098c107080 con 0x7f098c0601e0 2023-12-21T03:44:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.533+0000 7f0973fff700 1 -- 172.21.15.148:0/2484977137 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f097c0095f0 con 0x7f098c0601e0 2023-12-21T03:44:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.533+0000 7f09909ea700 1 -- 172.21.15.148:0/2484977137 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098c045ad0 con 0x7f098c0601e0 2023-12-21T03:44:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.533+0000 7f0973fff700 1 --2- 172.21.15.148:0/2484977137 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f097402ff60 0x7f0974032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.533+0000 7f0973fff700 1 -- 172.21.15.148:0/2484977137 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f097c040fc0 con 0x7f098c0601e0 2023-12-21T03:44:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.533+0000 7f098affd700 1 --2- 172.21.15.148:0/2484977137 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f097402ff60 0x7f0974032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.533+0000 7f098affd700 1 --2- 172.21.15.148:0/2484977137 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f097402ff60 0x7f0974032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f09800096a0 tx=0x7f0980006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.537+0000 7f0973fff700 1 -- 172.21.15.148:0/2484977137 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f097c014420 con 0x7f098c0601e0 2023-12-21T03:44:23.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.743+0000 7f09909ea700 1 -- 172.21.15.148:0/2484977137 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f098c107ed0 con 0x7f098c0601e0 2023-12-21T03:44:23.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.743+0000 7f0973fff700 1 -- 172.21.15.148:0/2484977137 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f097c010230 con 0x7f098c0601e0 2023-12-21T03:44:23.745 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:23.745 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:23.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.745+0000 7f09909ea700 1 -- 172.21.15.148:0/2484977137 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f097402ff60 msgr2=0x7f0974032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:23.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.745+0000 7f09909ea700 1 --2- 172.21.15.148:0/2484977137 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f097402ff60 0x7f0974032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f09800096a0 tx=0x7f0980006b40).stop 2023-12-21T03:44:23.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.745+0000 7f09909ea700 1 -- 172.21.15.148:0/2484977137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f098c0601e0 msgr2=0x7f098c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:23.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.745+0000 7f09909ea700 1 --2- 172.21.15.148:0/2484977137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f098c0601e0 0x7f098c106500 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f097c013510 tx=0x7f097c00b040).stop 2023-12-21T03:44:23.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.745+0000 7f09909ea700 1 -- 172.21.15.148:0/2484977137 shutdown_connections 2023-12-21T03:44:23.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.745+0000 7f09909ea700 1 --2- 172.21.15.148:0/2484977137 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f097402ff60 0x7f0974032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:23.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.745+0000 7f09909ea700 1 --2- 172.21.15.148:0/2484977137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f098c0601e0 0x7f098c106500 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:23.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.745+0000 7f09909ea700 1 -- 172.21.15.148:0/2484977137 >> 172.21.15.148:0/2484977137 conn(0x7f098c0f09a0 msgr2=0x7f098c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:23.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.745+0000 7f09909ea700 1 -- 172.21.15.148:0/2484977137 shutdown_connections 2023-12-21T03:44:23.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:23.745+0000 7f09909ea700 1 -- 172.21.15.148:0/2484977137 wait complete. 2023-12-21T03:44:23.747 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:24.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:23.744168+0000 mon.smithi078 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.148:0/2484977137' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:25.149 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:25.150 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:25.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:24.447211+0000 mgr.smithi078.zidbgq (mgr.14162) 154 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:26.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.233+0000 7fe528e85700 1 -- 172.21.15.148:0/852169799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5240f4740 msgr2=0x7fe5240f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:26.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.233+0000 7fe528e85700 1 --2- 172.21.15.148:0/852169799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5240f4740 0x7fe5240f4b60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fe518009fb0 tx=0x7fe518009b30).stop 2023-12-21T03:44:26.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.233+0000 7fe528e85700 1 -- 172.21.15.148:0/852169799 shutdown_connections 2023-12-21T03:44:26.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.233+0000 7fe528e85700 1 --2- 172.21.15.148:0/852169799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5240f4740 0x7fe5240f4b60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:26.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.233+0000 7fe528e85700 1 -- 172.21.15.148:0/852169799 >> 172.21.15.148:0/852169799 conn(0x7fe5240f0260 msgr2=0x7fe5240f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:26.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.233+0000 7fe528e85700 1 -- 172.21.15.148:0/852169799 shutdown_connections 2023-12-21T03:44:26.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.233+0000 7fe528e85700 1 -- 172.21.15.148:0/852169799 wait complete. 2023-12-21T03:44:26.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.234+0000 7fe528e85700 1 Processor -- start 2023-12-21T03:44:26.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.234+0000 7fe528e85700 1 -- start start 2023-12-21T03:44:26.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.234+0000 7fe528e85700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5240f4740 0x7fe5241066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:26.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.234+0000 7fe528e85700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe524106bc0 con 0x7fe5240f4740 2023-12-21T03:44:26.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.234+0000 7fe52259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5240f4740 0x7fe5241066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:26.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.234+0000 7fe52259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5240f4740 0x7fe5241066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:60922/0 (socket says 172.21.15.148:60922) 2023-12-21T03:44:26.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.234+0000 7fe52259c700 1 -- 172.21.15.148:0/1629669322 learned_addr learned my addr 172.21.15.148:0/1629669322 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:26.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.234+0000 7fe52259c700 1 -- 172.21.15.148:0/1629669322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe518005040 con 0x7fe5240f4740 2023-12-21T03:44:26.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.235+0000 7fe52259c700 1 --2- 172.21.15.148:0/1629669322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5240f4740 0x7fe5241066d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fe518013510 tx=0x7fe51800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:26.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.235+0000 7fe512ffd700 1 -- 172.21.15.148:0/1629669322 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe518009320 con 0x7fe5240f4740 2023-12-21T03:44:26.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.235+0000 7fe528e85700 1 -- 172.21.15.148:0/1629669322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe524106dc0 con 0x7fe5240f4740 2023-12-21T03:44:26.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.235+0000 7fe512ffd700 1 -- 172.21.15.148:0/1629669322 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe518013f00 con 0x7fe5240f4740 2023-12-21T03:44:26.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.235+0000 7fe512ffd700 1 -- 172.21.15.148:0/1629669322 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe518014d90 con 0x7fe5240f4740 2023-12-21T03:44:26.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.235+0000 7fe528e85700 1 -- 172.21.15.148:0/1629669322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe524107250 con 0x7fe5240f4740 2023-12-21T03:44:26.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.236+0000 7fe528e85700 1 -- 172.21.15.148:0/1629669322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe524045ad0 con 0x7fe5240f4740 2023-12-21T03:44:26.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.236+0000 7fe512ffd700 1 -- 172.21.15.148:0/1629669322 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5180095f0 con 0x7fe5240f4740 2023-12-21T03:44:26.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.236+0000 7fe512ffd700 1 --2- 172.21.15.148:0/1629669322 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe50c02ff60 0x7fe50c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:26.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.236+0000 7fe512ffd700 1 -- 172.21.15.148:0/1629669322 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe518041000 con 0x7fe5240f4740 2023-12-21T03:44:26.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.236+0000 7fe521d9b700 1 --2- 172.21.15.148:0/1629669322 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe50c02ff60 0x7fe50c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:26.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.236+0000 7fe521d9b700 1 --2- 172.21.15.148:0/1629669322 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe50c02ff60 0x7fe50c032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fe5140096a0 tx=0x7fe514006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:26.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.240+0000 7fe512ffd700 1 -- 172.21.15.148:0/1629669322 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe518014420 con 0x7fe5240f4740 2023-12-21T03:44:26.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.449+0000 7fe528e85700 1 -- 172.21.15.148:0/1629669322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe524108100 con 0x7fe5240f4740 2023-12-21T03:44:26.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.449+0000 7fe512ffd700 1 -- 172.21.15.148:0/1629669322 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe518010230 con 0x7fe5240f4740 2023-12-21T03:44:26.452 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:26.452 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.451+0000 7fe528e85700 1 -- 172.21.15.148:0/1629669322 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe50c02ff60 msgr2=0x7fe50c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.451+0000 7fe528e85700 1 --2- 172.21.15.148:0/1629669322 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe50c02ff60 0x7fe50c032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fe5140096a0 tx=0x7fe514006b40).stop 2023-12-21T03:44:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.451+0000 7fe528e85700 1 -- 172.21.15.148:0/1629669322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5240f4740 msgr2=0x7fe5241066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.451+0000 7fe528e85700 1 --2- 172.21.15.148:0/1629669322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5240f4740 0x7fe5241066d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fe518013510 tx=0x7fe51800b040).stop 2023-12-21T03:44:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.451+0000 7fe528e85700 1 -- 172.21.15.148:0/1629669322 shutdown_connections 2023-12-21T03:44:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.451+0000 7fe528e85700 1 --2- 172.21.15.148:0/1629669322 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe50c02ff60 0x7fe50c032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.451+0000 7fe528e85700 1 --2- 172.21.15.148:0/1629669322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5240f4740 0x7fe5241066d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.451+0000 7fe528e85700 1 -- 172.21.15.148:0/1629669322 >> 172.21.15.148:0/1629669322 conn(0x7fe5240f0260 msgr2=0x7fe5240f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.451+0000 7fe528e85700 1 -- 172.21.15.148:0/1629669322 shutdown_connections 2023-12-21T03:44:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:26.451+0000 7fe528e85700 1 -- 172.21.15.148:0/1629669322 wait complete. 2023-12-21T03:44:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:26.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:26.447554+0000 mgr.smithi078.zidbgq (mgr.14162) 155 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:26.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:26.451440+0000 mon.smithi078 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.148:0/1629669322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:27.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:27.824 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:28.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.869+0000 7f2a03870700 1 -- 172.21.15.148:0/2933444946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29fc0f5410 msgr2=0x7f29fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:28.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.869+0000 7f2a03870700 1 --2- 172.21.15.148:0/2933444946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29fc0f5410 0x7f29fc0f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f29ec009fb0 tx=0x7f29ec009b30).stop 2023-12-21T03:44:28.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.869+0000 7f2a03870700 1 -- 172.21.15.148:0/2933444946 shutdown_connections 2023-12-21T03:44:28.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.869+0000 7f2a03870700 1 --2- 172.21.15.148:0/2933444946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29fc0f5410 0x7f29fc0f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:28.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.869+0000 7f2a03870700 1 -- 172.21.15.148:0/2933444946 >> 172.21.15.148:0/2933444946 conn(0x7f29fc0f0b70 msgr2=0x7f29fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:28.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.869+0000 7f2a03870700 1 -- 172.21.15.148:0/2933444946 shutdown_connections 2023-12-21T03:44:28.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.869+0000 7f2a03870700 1 -- 172.21.15.148:0/2933444946 wait complete. 2023-12-21T03:44:28.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.870+0000 7f2a03870700 1 Processor -- start 2023-12-21T03:44:28.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.870+0000 7f2a03870700 1 -- start start 2023-12-21T03:44:28.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.870+0000 7f2a03870700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29fc0f5410 0x7f29fc106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:28.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.870+0000 7f2a03870700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc106c10 con 0x7f29fc0f5410 2023-12-21T03:44:28.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.870+0000 7f2a0160c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29fc0f5410 0x7f29fc106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:28.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.870+0000 7f2a0160c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29fc0f5410 0x7f29fc106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:60940/0 (socket says 172.21.15.148:60940) 2023-12-21T03:44:28.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.870+0000 7f2a0160c700 1 -- 172.21.15.148:0/3309917759 learned_addr learned my addr 172.21.15.148:0/3309917759 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:28.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.870+0000 7f2a0160c700 1 -- 172.21.15.148:0/3309917759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec005040 con 0x7f29fc0f5410 2023-12-21T03:44:28.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.870+0000 7f2a0160c700 1 --2- 172.21.15.148:0/3309917759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29fc0f5410 0x7f29fc106720 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f29ec013510 tx=0x7f29ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:28.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.870+0000 7f29f1ffb700 1 -- 172.21.15.148:0/3309917759 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29ec009320 con 0x7f29fc0f5410 2023-12-21T03:44:28.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.871+0000 7f2a03870700 1 -- 172.21.15.148:0/3309917759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29fc106e10 con 0x7f29fc0f5410 2023-12-21T03:44:28.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.871+0000 7f29f1ffb700 1 -- 172.21.15.148:0/3309917759 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29ec013f00 con 0x7f29fc0f5410 2023-12-21T03:44:28.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.871+0000 7f29f1ffb700 1 -- 172.21.15.148:0/3309917759 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29ec014d90 con 0x7f29fc0f5410 2023-12-21T03:44:28.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.871+0000 7f2a03870700 1 -- 172.21.15.148:0/3309917759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29fc1072a0 con 0x7f29fc0f5410 2023-12-21T03:44:28.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.871+0000 7f29f1ffb700 1 -- 172.21.15.148:0/3309917759 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29ec0263a0 con 0x7f29fc0f5410 2023-12-21T03:44:28.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.872+0000 7f2a03870700 1 -- 172.21.15.148:0/3309917759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29fc045ad0 con 0x7f29fc0f5410 2023-12-21T03:44:28.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.872+0000 7f29f1ffb700 1 --2- 172.21.15.148:0/3309917759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f29e802ffb0 0x7f29e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:28.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.872+0000 7f29f1ffb700 1 -- 172.21.15.148:0/3309917759 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29ec041100 con 0x7f29fc0f5410 2023-12-21T03:44:28.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.872+0000 7f2a00e0b700 1 --2- 172.21.15.148:0/3309917759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f29e802ffb0 0x7f29e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:28.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.872+0000 7f2a00e0b700 1 --2- 172.21.15.148:0/3309917759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f29e802ffb0 0x7f29e8032470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f29f80096a0 tx=0x7f29f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:28.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:28.876+0000 7f29f1ffb700 1 -- 172.21.15.148:0/3309917759 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29ec0095f0 con 0x7f29fc0f5410 2023-12-21T03:44:29.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.079+0000 7f2a03870700 1 -- 172.21.15.148:0/3309917759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29fc1080f0 con 0x7f29fc0f5410 2023-12-21T03:44:29.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.080+0000 7f29f1ffb700 1 -- 172.21.15.148:0/3309917759 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f29ec01db70 con 0x7f29fc0f5410 2023-12-21T03:44:29.083 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:29.083 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:29.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.082+0000 7f2a03870700 1 -- 172.21.15.148:0/3309917759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f29e802ffb0 msgr2=0x7f29e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:29.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.082+0000 7f2a03870700 1 --2- 172.21.15.148:0/3309917759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f29e802ffb0 0x7f29e8032470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f29f80096a0 tx=0x7f29f8006b40).stop 2023-12-21T03:44:29.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.082+0000 7f2a03870700 1 -- 172.21.15.148:0/3309917759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29fc0f5410 msgr2=0x7f29fc106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:29.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.082+0000 7f2a03870700 1 --2- 172.21.15.148:0/3309917759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29fc0f5410 0x7f29fc106720 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f29ec013510 tx=0x7f29ec00b040).stop 2023-12-21T03:44:29.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.082+0000 7f2a03870700 1 -- 172.21.15.148:0/3309917759 shutdown_connections 2023-12-21T03:44:29.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.082+0000 7f2a03870700 1 --2- 172.21.15.148:0/3309917759 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f29e802ffb0 0x7f29e8032470 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:29.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.082+0000 7f2a03870700 1 --2- 172.21.15.148:0/3309917759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29fc0f5410 0x7f29fc106720 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:29.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.082+0000 7f2a03870700 1 -- 172.21.15.148:0/3309917759 >> 172.21.15.148:0/3309917759 conn(0x7f29fc0f0b70 msgr2=0x7f29fc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:29.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.082+0000 7f2a03870700 1 -- 172.21.15.148:0/3309917759 shutdown_connections 2023-12-21T03:44:29.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:29.082+0000 7f2a03870700 1 -- 172.21.15.148:0/3309917759 wait complete. 2023-12-21T03:44:29.085 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:28.447880+0000 mgr.smithi078.zidbgq (mgr.14162) 156 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:29.082019+0000 mon.smithi078 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.148:0/3309917759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:30.487 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:30.487 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:30.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:30.448208+0000 mgr.smithi078.zidbgq (mgr.14162) 157 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:31.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.611+0000 7f1474a4c700 1 -- 172.21.15.148:0/2678347446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14700f51a0 msgr2=0x7f14700f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:31.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.611+0000 7f1474a4c700 1 --2- 172.21.15.148:0/2678347446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14700f51a0 0x7f14700f55c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f1460009fb0 tx=0x7f1460009b30).stop 2023-12-21T03:44:31.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.611+0000 7f1474a4c700 1 -- 172.21.15.148:0/2678347446 shutdown_connections 2023-12-21T03:44:31.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.611+0000 7f1474a4c700 1 --2- 172.21.15.148:0/2678347446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14700f51a0 0x7f14700f55c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:31.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.611+0000 7f1474a4c700 1 -- 172.21.15.148:0/2678347446 >> 172.21.15.148:0/2678347446 conn(0x7f14700f09a0 msgr2=0x7f14700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:31.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.611+0000 7f1474a4c700 1 -- 172.21.15.148:0/2678347446 shutdown_connections 2023-12-21T03:44:31.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.611+0000 7f1474a4c700 1 -- 172.21.15.148:0/2678347446 wait complete. 2023-12-21T03:44:31.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.611+0000 7f1474a4c700 1 Processor -- start 2023-12-21T03:44:31.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.611+0000 7f1474a4c700 1 -- start start 2023-12-21T03:44:31.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f1474a4c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14700f51a0 0x7f14701091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:31.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f1474a4c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470060220 con 0x7f14700f51a0 2023-12-21T03:44:31.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f146f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14700f51a0 0x7f14701091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:31.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f146f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14700f51a0 0x7f14701091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:45282/0 (socket says 172.21.15.148:45282) 2023-12-21T03:44:31.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f146f7fe700 1 -- 172.21.15.148:0/3427034091 learned_addr learned my addr 172.21.15.148:0/3427034091 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:31.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f146f7fe700 1 -- 172.21.15.148:0/3427034091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1460005040 con 0x7f14700f51a0 2023-12-21T03:44:31.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f146f7fe700 1 --2- 172.21.15.148:0/3427034091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14700f51a0 0x7f14701091d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1460013510 tx=0x7f146000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:31.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f1457fff700 1 -- 172.21.15.148:0/3427034091 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1460009320 con 0x7f14700f51a0 2023-12-21T03:44:31.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f1457fff700 1 -- 172.21.15.148:0/3427034091 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1460013f00 con 0x7f14700f51a0 2023-12-21T03:44:31.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f1474a4c700 1 -- 172.21.15.148:0/3427034091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1470060420 con 0x7f14700f51a0 2023-12-21T03:44:31.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f1457fff700 1 -- 172.21.15.148:0/3427034091 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1460014dc0 con 0x7f14700f51a0 2023-12-21T03:44:31.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.612+0000 7f1474a4c700 1 -- 172.21.15.148:0/3427034091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1470109880 con 0x7f14700f51a0 2023-12-21T03:44:31.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.613+0000 7f1457fff700 1 -- 172.21.15.148:0/3427034091 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14600095f0 con 0x7f14700f51a0 2023-12-21T03:44:31.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.613+0000 7f1474a4c700 1 -- 172.21.15.148:0/3427034091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1470059100 con 0x7f14700f51a0 2023-12-21T03:44:31.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.613+0000 7f1457fff700 1 --2- 172.21.15.148:0/3427034091 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f145802ff60 0x7f1458032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:31.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.613+0000 7f1457fff700 1 -- 172.21.15.148:0/3427034091 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14600410c0 con 0x7f14700f51a0 2023-12-21T03:44:31.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.614+0000 7f146effd700 1 --2- 172.21.15.148:0/3427034091 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f145802ff60 0x7f1458032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:31.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.614+0000 7f146effd700 1 --2- 172.21.15.148:0/3427034091 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f145802ff60 0x7f1458032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f14640096a0 tx=0x7f1464006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:31.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.618+0000 7f1457fff700 1 -- 172.21.15.148:0/3427034091 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f146001c790 con 0x7f14700f51a0 2023-12-21T03:44:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:30.545013+0000 mon.smithi078 (mon.0) 232 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:44:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:30.552121+0000 mon.smithi078 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:44:31.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.825+0000 7f1474a4c700 1 -- 172.21.15.148:0/3427034091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1470045ad0 con 0x7f14700f51a0 2023-12-21T03:44:31.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.826+0000 7f1457fff700 1 -- 172.21.15.148:0/3427034091 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1460010230 con 0x7f14700f51a0 2023-12-21T03:44:31.828 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:31.828 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:31.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.827+0000 7f1474a4c700 1 -- 172.21.15.148:0/3427034091 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f145802ff60 msgr2=0x7f1458032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:31.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.827+0000 7f1474a4c700 1 --2- 172.21.15.148:0/3427034091 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f145802ff60 0x7f1458032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f14640096a0 tx=0x7f1464006b40).stop 2023-12-21T03:44:31.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.827+0000 7f1474a4c700 1 -- 172.21.15.148:0/3427034091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14700f51a0 msgr2=0x7f14701091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:31.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.827+0000 7f1474a4c700 1 --2- 172.21.15.148:0/3427034091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14700f51a0 0x7f14701091d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1460013510 tx=0x7f146000b040).stop 2023-12-21T03:44:31.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.827+0000 7f1474a4c700 1 -- 172.21.15.148:0/3427034091 shutdown_connections 2023-12-21T03:44:31.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.827+0000 7f1474a4c700 1 --2- 172.21.15.148:0/3427034091 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f145802ff60 0x7f1458032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:31.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.827+0000 7f1474a4c700 1 --2- 172.21.15.148:0/3427034091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14700f51a0 0x7f14701091d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:31.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.827+0000 7f1474a4c700 1 -- 172.21.15.148:0/3427034091 >> 172.21.15.148:0/3427034091 conn(0x7f14700f09a0 msgr2=0x7f14700f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:31.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.827+0000 7f1474a4c700 1 -- 172.21.15.148:0/3427034091 shutdown_connections 2023-12-21T03:44:31.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:31.827+0000 7f1474a4c700 1 -- 172.21.15.148:0/3427034091 wait complete. 2023-12-21T03:44:31.831 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:32.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:31.827548+0000 mon.smithi078 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.148:0/3427034091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:32.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:32.448568+0000 mgr.smithi078.zidbgq (mgr.14162) 158 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:33.192 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:33.193 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:34.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.377+0000 7ff910c05700 1 -- 172.21.15.148:0/2334963185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff90c0f62a0 msgr2=0x7ff90c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:34.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.377+0000 7ff910c05700 1 --2- 172.21.15.148:0/2334963185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff90c0f62a0 0x7ff90c0f66c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff8f4009fb0 tx=0x7ff8f4009b30).stop 2023-12-21T03:44:34.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.377+0000 7ff910c05700 1 -- 172.21.15.148:0/2334963185 shutdown_connections 2023-12-21T03:44:34.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.377+0000 7ff910c05700 1 --2- 172.21.15.148:0/2334963185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff90c0f62a0 0x7ff90c0f66c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:34.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.377+0000 7ff910c05700 1 -- 172.21.15.148:0/2334963185 >> 172.21.15.148:0/2334963185 conn(0x7ff90c0f1aa0 msgr2=0x7ff90c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.377+0000 7ff910c05700 1 -- 172.21.15.148:0/2334963185 shutdown_connections 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.377+0000 7ff910c05700 1 -- 172.21.15.148:0/2334963185 wait complete. 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.378+0000 7ff910c05700 1 Processor -- start 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.378+0000 7ff910c05700 1 -- start start 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.378+0000 7ff910c05700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff90c0f62a0 0x7ff90c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.378+0000 7ff910c05700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c1069a0 con 0x7ff90c0f62a0 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.378+0000 7ff90b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff90c0f62a0 0x7ff90c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.378+0000 7ff90b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff90c0f62a0 0x7ff90c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:45312/0 (socket says 172.21.15.148:45312) 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.378+0000 7ff90b7fe700 1 -- 172.21.15.148:0/2586118636 learned_addr learned my addr 172.21.15.148:0/2586118636 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.378+0000 7ff90b7fe700 1 -- 172.21.15.148:0/2586118636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f4005040 con 0x7ff90c0f62a0 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.379+0000 7ff90b7fe700 1 --2- 172.21.15.148:0/2586118636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff90c0f62a0 0x7ff90c1064b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7ff8f4013510 tx=0x7ff8f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.379+0000 7ff903fff700 1 -- 172.21.15.148:0/2586118636 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f4009320 con 0x7ff90c0f62a0 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.379+0000 7ff910c05700 1 -- 172.21.15.148:0/2586118636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90c106ba0 con 0x7ff90c0f62a0 2023-12-21T03:44:34.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.379+0000 7ff903fff700 1 -- 172.21.15.148:0/2586118636 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8f4013f00 con 0x7ff90c0f62a0 2023-12-21T03:44:34.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.379+0000 7ff903fff700 1 -- 172.21.15.148:0/2586118636 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f4014dc0 con 0x7ff90c0f62a0 2023-12-21T03:44:34.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.379+0000 7ff910c05700 1 -- 172.21.15.148:0/2586118636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90c107030 con 0x7ff90c0f62a0 2023-12-21T03:44:34.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.380+0000 7ff903fff700 1 -- 172.21.15.148:0/2586118636 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8f401d3d0 con 0x7ff90c0f62a0 2023-12-21T03:44:34.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.380+0000 7ff903fff700 1 --2- 172.21.15.148:0/2586118636 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff8f802ff60 0x7ff8f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:34.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.380+0000 7ff903fff700 1 -- 172.21.15.148:0/2586118636 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8f4041010 con 0x7ff90c0f62a0 2023-12-21T03:44:34.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.380+0000 7ff910c05700 1 -- 172.21.15.148:0/2586118636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90c059100 con 0x7ff90c0f62a0 2023-12-21T03:44:34.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.380+0000 7ff90affd700 1 --2- 172.21.15.148:0/2586118636 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff8f802ff60 0x7ff8f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:34.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.380+0000 7ff90affd700 1 --2- 172.21.15.148:0/2586118636 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff8f802ff60 0x7ff8f8032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff8fc0096a0 tx=0x7ff8fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:34.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.385+0000 7ff903fff700 1 -- 172.21.15.148:0/2586118636 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8f4012900 con 0x7ff90c0f62a0 2023-12-21T03:44:34.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.593+0000 7ff910c05700 1 -- 172.21.15.148:0/2586118636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff90c0fbce0 con 0x7ff90c0f62a0 2023-12-21T03:44:34.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.594+0000 7ff903fff700 1 -- 172.21.15.148:0/2586118636 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff8f402b020 con 0x7ff90c0f62a0 2023-12-21T03:44:34.596 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:34.596 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.596+0000 7ff910c05700 1 -- 172.21.15.148:0/2586118636 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff8f802ff60 msgr2=0x7ff8f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.596+0000 7ff910c05700 1 --2- 172.21.15.148:0/2586118636 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff8f802ff60 0x7ff8f8032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff8fc0096a0 tx=0x7ff8fc006b40).stop 2023-12-21T03:44:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.596+0000 7ff910c05700 1 -- 172.21.15.148:0/2586118636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff90c0f62a0 msgr2=0x7ff90c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.596+0000 7ff910c05700 1 --2- 172.21.15.148:0/2586118636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff90c0f62a0 0x7ff90c1064b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7ff8f4013510 tx=0x7ff8f400b040).stop 2023-12-21T03:44:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.596+0000 7ff910c05700 1 -- 172.21.15.148:0/2586118636 shutdown_connections 2023-12-21T03:44:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.596+0000 7ff910c05700 1 --2- 172.21.15.148:0/2586118636 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff8f802ff60 0x7ff8f8032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.596+0000 7ff910c05700 1 --2- 172.21.15.148:0/2586118636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff90c0f62a0 0x7ff90c1064b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.596+0000 7ff910c05700 1 -- 172.21.15.148:0/2586118636 >> 172.21.15.148:0/2586118636 conn(0x7ff90c0f1aa0 msgr2=0x7ff90c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.596+0000 7ff910c05700 1 -- 172.21.15.148:0/2586118636 shutdown_connections 2023-12-21T03:44:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:34.596+0000 7ff910c05700 1 -- 172.21.15.148:0/2586118636 wait complete. 2023-12-21T03:44:34.599 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:35.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:34.448913+0000 mgr.smithi078.zidbgq (mgr.14162) 159 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:35.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:34.595902+0000 mon.smithi078 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.148:0/2586118636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:35.978 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:35.978 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:36.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:36 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:35.956808+0000 mon.smithi078 (mon.0) 236 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:44:36.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:36 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:36.212226+0000 mon.smithi078 (mon.0) 237 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:44:37.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.201+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3546709899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9480f5410 msgr2=0x7fc9480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:37.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.201+0000 7fc94fcc2700 1 --2- 172.21.15.148:0/3546709899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9480f5410 0x7fc9480f5830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fc938009fb0 tx=0x7fc938009b30).stop 2023-12-21T03:44:37.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.201+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3546709899 shutdown_connections 2023-12-21T03:44:37.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.201+0000 7fc94fcc2700 1 --2- 172.21.15.148:0/3546709899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9480f5410 0x7fc9480f5830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:37.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.201+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3546709899 >> 172.21.15.148:0/3546709899 conn(0x7fc9480f0b70 msgr2=0x7fc9480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:37.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.201+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3546709899 shutdown_connections 2023-12-21T03:44:37.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.201+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3546709899 wait complete. 2023-12-21T03:44:37.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.202+0000 7fc94fcc2700 1 Processor -- start 2023-12-21T03:44:37.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.202+0000 7fc94fcc2700 1 -- start start 2023-12-21T03:44:37.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.202+0000 7fc94fcc2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9480f5410 0x7fc948105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:37.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.202+0000 7fc94fcc2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc948105b20 con 0x7fc9480f5410 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.202+0000 7fc94da5e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9480f5410 0x7fc948105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.202+0000 7fc94da5e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9480f5410 0x7fc948105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:45334/0 (socket says 172.21.15.148:45334) 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.202+0000 7fc94da5e700 1 -- 172.21.15.148:0/3553119537 learned_addr learned my addr 172.21.15.148:0/3553119537 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc94da5e700 1 -- 172.21.15.148:0/3553119537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc938005040 con 0x7fc9480f5410 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc94da5e700 1 --2- 172.21.15.148:0/3553119537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9480f5410 0x7fc948105630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fc938013510 tx=0x7fc93800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc93e7fc700 1 -- 172.21.15.148:0/3553119537 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc938009320 con 0x7fc9480f5410 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc93e7fc700 1 -- 172.21.15.148:0/3553119537 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc938013f00 con 0x7fc9480f5410 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3553119537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc948105d20 con 0x7fc9480f5410 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc93e7fc700 1 -- 172.21.15.148:0/3553119537 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc938014d90 con 0x7fc9480f5410 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3553119537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9481061b0 con 0x7fc9480f5410 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc93e7fc700 1 -- 172.21.15.148:0/3553119537 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9380263a0 con 0x7fc9480f5410 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3553119537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc948045ad0 con 0x7fc9480f5410 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc93e7fc700 1 --2- 172.21.15.148:0/3553119537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc93402ff60 0x7fc934032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:37.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc93e7fc700 1 -- 172.21.15.148:0/3553119537 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9380413b0 con 0x7fc9480f5410 2023-12-21T03:44:37.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc94d25d700 1 --2- 172.21.15.148:0/3553119537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc93402ff60 0x7fc934032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:37.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.204+0000 7fc94d25d700 1 --2- 172.21.15.148:0/3553119537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc93402ff60 0x7fc934032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc9440096a0 tx=0x7fc944006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:37.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.208+0000 7fc93e7fc700 1 -- 172.21.15.148:0/3553119537 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9380126f0 con 0x7fc9480f5410 2023-12-21T03:44:37.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.425+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3553119537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9480fae60 con 0x7fc9480f5410 2023-12-21T03:44:37.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.425+0000 7fc93e7fc700 1 -- 172.21.15.148:0/3553119537 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc938012330 con 0x7fc9480f5410 2023-12-21T03:44:37.426 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:37.426 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:37.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.427+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3553119537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc93402ff60 msgr2=0x7fc934032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:37.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.427+0000 7fc94fcc2700 1 --2- 172.21.15.148:0/3553119537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc93402ff60 0x7fc934032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc9440096a0 tx=0x7fc944006b40).stop 2023-12-21T03:44:37.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.427+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3553119537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9480f5410 msgr2=0x7fc948105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:37.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.427+0000 7fc94fcc2700 1 --2- 172.21.15.148:0/3553119537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9480f5410 0x7fc948105630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fc938013510 tx=0x7fc93800b040).stop 2023-12-21T03:44:37.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.427+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3553119537 shutdown_connections 2023-12-21T03:44:37.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.427+0000 7fc94fcc2700 1 --2- 172.21.15.148:0/3553119537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc93402ff60 0x7fc934032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:37.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.427+0000 7fc94fcc2700 1 --2- 172.21.15.148:0/3553119537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9480f5410 0x7fc948105630 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:37.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.427+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3553119537 >> 172.21.15.148:0/3553119537 conn(0x7fc9480f0b70 msgr2=0x7fc9480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:37.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.427+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3553119537 shutdown_connections 2023-12-21T03:44:37.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:37.427+0000 7fc94fcc2700 1 -- 172.21.15.148:0/3553119537 wait complete. 2023-12-21T03:44:37.429 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:44:35.956142+0000 mgr.smithi078.zidbgq (mgr.14162) 160 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi078 ... 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:42:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:43:20Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:43:58Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:44:37.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('uid/gid not found') 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: uid/gid not found 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: yield (conn, connr) 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:44:37.568 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: code, '\n'.join(err))) 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi078 ... 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:42:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:43:20Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:43:58Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:44:37.569 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:44:37.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-21T03:44:37.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-21T03:44:37.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-12-21T03:44:37.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-12-21T03:44:37.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-21T03:44:37.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('uid/gid not found') 2023-12-21T03:44:37.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: uid/gid not found 2023-12-21T03:44:37.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:36.449159+0000 mgr.smithi078.zidbgq (mgr.14162) 161 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:37.570 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:36.519837+0000 mon.smithi078 (mon.0) 238 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:44:38.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:37.426014+0000 mon.smithi078 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.148:0/3553119537' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:38.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:37.670423+0000 mon.smithi078 (mon.0) 240 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:44:38.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:38.832 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:39.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:38.449427+0000 mgr.smithi078.zidbgq (mgr.14162) 162 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:39.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:39.236220+0000 mon.smithi078 (mon.0) 241 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:44:40.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.057+0000 7f85a57b3700 1 -- 172.21.15.148:0/314335983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a00f47c0 msgr2=0x7f85a00f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:40.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.057+0000 7f85a57b3700 1 --2- 172.21.15.148:0/314335983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a00f47c0 0x7f85a00f4be0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f8594009fb0 tx=0x7f8594009b30).stop 2023-12-21T03:44:40.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.057+0000 7f85a57b3700 1 -- 172.21.15.148:0/314335983 shutdown_connections 2023-12-21T03:44:40.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.057+0000 7f85a57b3700 1 --2- 172.21.15.148:0/314335983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a00f47c0 0x7f85a00f4be0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:40.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.057+0000 7f85a57b3700 1 -- 172.21.15.148:0/314335983 >> 172.21.15.148:0/314335983 conn(0x7f85a00f02e0 msgr2=0x7f85a00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:40.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.057+0000 7f85a57b3700 1 -- 172.21.15.148:0/314335983 shutdown_connections 2023-12-21T03:44:40.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.057+0000 7f85a57b3700 1 -- 172.21.15.148:0/314335983 wait complete. 2023-12-21T03:44:40.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.058+0000 7f85a57b3700 1 Processor -- start 2023-12-21T03:44:40.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.058+0000 7f85a57b3700 1 -- start start 2023-12-21T03:44:40.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.058+0000 7f85a57b3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a00f47c0 0x7f85a0106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:40.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.058+0000 7f85a57b3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a0106bf0 con 0x7f85a00f47c0 2023-12-21T03:44:40.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.058+0000 7f859effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a00f47c0 0x7f85a0106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:40.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.059+0000 7f859effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a00f47c0 0x7f85a0106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35600/0 (socket says 172.21.15.148:35600) 2023-12-21T03:44:40.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.059+0000 7f859effd700 1 -- 172.21.15.148:0/2006138158 learned_addr learned my addr 172.21.15.148:0/2006138158 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:40.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.059+0000 7f859effd700 1 -- 172.21.15.148:0/2006138158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8594005040 con 0x7f85a00f47c0 2023-12-21T03:44:40.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.059+0000 7f859effd700 1 --2- 172.21.15.148:0/2006138158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a00f47c0 0x7f85a0106700 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f8594013510 tx=0x7f859400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:40.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.059+0000 7f85877fe700 1 -- 172.21.15.148:0/2006138158 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8594009320 con 0x7f85a00f47c0 2023-12-21T03:44:40.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.059+0000 7f85a57b3700 1 -- 172.21.15.148:0/2006138158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a0106df0 con 0x7f85a00f47c0 2023-12-21T03:44:40.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.059+0000 7f85877fe700 1 -- 172.21.15.148:0/2006138158 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8594013f00 con 0x7f85a00f47c0 2023-12-21T03:44:40.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.059+0000 7f85877fe700 1 -- 172.21.15.148:0/2006138158 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8594014d90 con 0x7f85a00f47c0 2023-12-21T03:44:40.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.059+0000 7f85a57b3700 1 -- 172.21.15.148:0/2006138158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a0107280 con 0x7f85a00f47c0 2023-12-21T03:44:40.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.060+0000 7f85877fe700 1 -- 172.21.15.148:0/2006138158 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85940095f0 con 0x7f85a00f47c0 2023-12-21T03:44:40.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.060+0000 7f85a57b3700 1 -- 172.21.15.148:0/2006138158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a0045ad0 con 0x7f85a00f47c0 2023-12-21T03:44:40.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.060+0000 7f85877fe700 1 --2- 172.21.15.148:0/2006138158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f858802ff60 0x7f8588032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:40.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.060+0000 7f85877fe700 1 -- 172.21.15.148:0/2006138158 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85940410f0 con 0x7f85a00f47c0 2023-12-21T03:44:40.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.061+0000 7f859e7fc700 1 --2- 172.21.15.148:0/2006138158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f858802ff60 0x7f8588032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:40.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.061+0000 7f859e7fc700 1 --2- 172.21.15.148:0/2006138158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f858802ff60 0x7f8588032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f85900096a0 tx=0x7f8590006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:40.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.065+0000 7f85877fe700 1 -- 172.21.15.148:0/2006138158 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8594014420 con 0x7f85a00f47c0 2023-12-21T03:44:40.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.279+0000 7f85a57b3700 1 -- 172.21.15.148:0/2006138158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85a0108130 con 0x7f85a00f47c0 2023-12-21T03:44:40.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.280+0000 7f85877fe700 1 -- 172.21.15.148:0/2006138158 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8594010230 con 0x7f85a00f47c0 2023-12-21T03:44:40.281 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:40.282 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:40.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.282+0000 7f85a57b3700 1 -- 172.21.15.148:0/2006138158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f858802ff60 msgr2=0x7f8588032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:40.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.282+0000 7f85a57b3700 1 --2- 172.21.15.148:0/2006138158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f858802ff60 0x7f8588032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f85900096a0 tx=0x7f8590006b40).stop 2023-12-21T03:44:40.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.282+0000 7f85a57b3700 1 -- 172.21.15.148:0/2006138158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a00f47c0 msgr2=0x7f85a0106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:40.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.282+0000 7f85a57b3700 1 --2- 172.21.15.148:0/2006138158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a00f47c0 0x7f85a0106700 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f8594013510 tx=0x7f859400b040).stop 2023-12-21T03:44:40.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.282+0000 7f85a57b3700 1 -- 172.21.15.148:0/2006138158 shutdown_connections 2023-12-21T03:44:40.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.282+0000 7f85a57b3700 1 --2- 172.21.15.148:0/2006138158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f858802ff60 0x7f8588032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:40.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.282+0000 7f85a57b3700 1 --2- 172.21.15.148:0/2006138158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a00f47c0 0x7f85a0106700 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:40.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.282+0000 7f85a57b3700 1 -- 172.21.15.148:0/2006138158 >> 172.21.15.148:0/2006138158 conn(0x7f85a00f02e0 msgr2=0x7f85a00f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:40.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.282+0000 7f85a57b3700 1 -- 172.21.15.148:0/2006138158 shutdown_connections 2023-12-21T03:44:40.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:40.282+0000 7f85a57b3700 1 -- 172.21.15.148:0/2006138158 wait complete. 2023-12-21T03:44:40.284 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:41.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:40.224524+0000 mon.smithi078 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:44:41.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:40.225397+0000 mgr.smithi078.zidbgq (mgr.14162) 163 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:41.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:40.230489+0000 mon.smithi078 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:44:41.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:40.231740+0000 mon.smithi078 (mon.0) 244 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi148", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-21T03:44:41.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:40.233736+0000 mon.smithi078 (mon.0) 245 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi148", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-21T03:44:41.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:40.234536+0000 mon.smithi078 (mon.0) 246 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:44:41.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:44:40.235200+0000 mgr.smithi078.zidbgq (mgr.14162) 164 : cephadm [INF] Deploying daemon crash.smithi148 on smithi148 2023-12-21T03:44:41.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:40.280656+0000 mon.smithi078 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.148:0/2006138158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:41.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:41.680 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:42.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:41.222715+0000 mon.smithi078 (mon.0) 248 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-12-21T03:44:42.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:42.133327+0000 mon.smithi078 (mon.0) 249 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:44:42.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:42.134759+0000 mon.smithi078 (mon.0) 250 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi148.tgpyxd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-21T03:44:42.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:42.136772+0000 mon.smithi078 (mon.0) 251 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi148.tgpyxd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-21T03:44:42.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:42.137600+0000 mon.smithi078 (mon.0) 252 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-21T03:44:42.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:42.138479+0000 mon.smithi078 (mon.0) 253 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:44:42.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.985+0000 7f0983bb3700 1 -- 172.21.15.148:0/20966401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f097c0f5410 msgr2=0x7f097c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:42.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.985+0000 7f0983bb3700 1 --2- 172.21.15.148:0/20966401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f097c0f5410 0x7f097c0f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f096c009fb0 tx=0x7f096c009b30).stop 2023-12-21T03:44:42.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.985+0000 7f0983bb3700 1 -- 172.21.15.148:0/20966401 shutdown_connections 2023-12-21T03:44:42.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.985+0000 7f0983bb3700 1 --2- 172.21.15.148:0/20966401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f097c0f5410 0x7f097c0f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:42.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.985+0000 7f0983bb3700 1 -- 172.21.15.148:0/20966401 >> 172.21.15.148:0/20966401 conn(0x7f097c0f0b70 msgr2=0x7f097c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:42.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.986+0000 7f0983bb3700 1 -- 172.21.15.148:0/20966401 shutdown_connections 2023-12-21T03:44:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.986+0000 7f0983bb3700 1 -- 172.21.15.148:0/20966401 wait complete. 2023-12-21T03:44:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.986+0000 7f0983bb3700 1 Processor -- start 2023-12-21T03:44:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.986+0000 7f0983bb3700 1 -- start start 2023-12-21T03:44:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.986+0000 7f0983bb3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f097c0f5410 0x7f097c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.986+0000 7f0983bb3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c106c10 con 0x7f097c0f5410 2023-12-21T03:44:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.987+0000 7f098194f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f097c0f5410 0x7f097c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.987+0000 7f098194f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f097c0f5410 0x7f097c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35628/0 (socket says 172.21.15.148:35628) 2023-12-21T03:44:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.987+0000 7f098194f700 1 -- 172.21.15.148:0/869405878 learned_addr learned my addr 172.21.15.148:0/869405878 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.987+0000 7f098194f700 1 -- 172.21.15.148:0/869405878 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c005040 con 0x7f097c0f5410 2023-12-21T03:44:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.987+0000 7f098194f700 1 --2- 172.21.15.148:0/869405878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f097c0f5410 0x7f097c106720 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f096c013510 tx=0x7f096c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.987+0000 7f09727fc700 1 -- 172.21.15.148:0/869405878 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f096c009320 con 0x7f097c0f5410 2023-12-21T03:44:42.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.987+0000 7f0983bb3700 1 -- 172.21.15.148:0/869405878 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097c106e10 con 0x7f097c0f5410 2023-12-21T03:44:42.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.987+0000 7f09727fc700 1 -- 172.21.15.148:0/869405878 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f096c013f00 con 0x7f097c0f5410 2023-12-21T03:44:42.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.987+0000 7f09727fc700 1 -- 172.21.15.148:0/869405878 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f096c014d90 con 0x7f097c0f5410 2023-12-21T03:44:42.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.987+0000 7f0983bb3700 1 -- 172.21.15.148:0/869405878 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097c1072a0 con 0x7f097c0f5410 2023-12-21T03:44:42.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.988+0000 7f09727fc700 1 -- 172.21.15.148:0/869405878 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f096c014420 con 0x7f097c0f5410 2023-12-21T03:44:42.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.988+0000 7f0983bb3700 1 -- 172.21.15.148:0/869405878 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097c045ad0 con 0x7f097c0f5410 2023-12-21T03:44:42.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.988+0000 7f09727fc700 1 --2- 172.21.15.148:0/869405878 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f096802ffb0 0x7f0968032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:42.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.988+0000 7f09727fc700 1 -- 172.21.15.148:0/869405878 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f096c0410c0 con 0x7f097c0f5410 2023-12-21T03:44:42.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.988+0000 7f098114e700 1 --2- 172.21.15.148:0/869405878 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f096802ffb0 0x7f0968032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:42.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.989+0000 7f098114e700 1 --2- 172.21.15.148:0/869405878 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f096802ffb0 0x7f0968032470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f09780096a0 tx=0x7f0978006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:42.993+0000 7f09727fc700 1 -- 172.21.15.148:0/869405878 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f096c0095f0 con 0x7f097c0f5410 2023-12-21T03:44:43.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.218+0000 7f0983bb3700 1 -- 172.21.15.148:0/869405878 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f097c108150 con 0x7f097c0f5410 2023-12-21T03:44:43.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.219+0000 7f09727fc700 1 -- 172.21.15.148:0/869405878 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f096c01db70 con 0x7f097c0f5410 2023-12-21T03:44:43.220 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:43.221 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:38:38.889362Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-21T03:44:43.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.221+0000 7f0983bb3700 1 -- 172.21.15.148:0/869405878 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f096802ffb0 msgr2=0x7f0968032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:43.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.221+0000 7f0983bb3700 1 --2- 172.21.15.148:0/869405878 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f096802ffb0 0x7f0968032470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f09780096a0 tx=0x7f0978006b40).stop 2023-12-21T03:44:43.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.221+0000 7f0983bb3700 1 -- 172.21.15.148:0/869405878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f097c0f5410 msgr2=0x7f097c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:43.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.221+0000 7f0983bb3700 1 --2- 172.21.15.148:0/869405878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f097c0f5410 0x7f097c106720 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f096c013510 tx=0x7f096c00b040).stop 2023-12-21T03:44:43.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.221+0000 7f0983bb3700 1 -- 172.21.15.148:0/869405878 shutdown_connections 2023-12-21T03:44:43.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.221+0000 7f0983bb3700 1 --2- 172.21.15.148:0/869405878 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f096802ffb0 0x7f0968032470 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:43.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.221+0000 7f0983bb3700 1 --2- 172.21.15.148:0/869405878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f097c0f5410 0x7f097c106720 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:43.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.221+0000 7f0983bb3700 1 -- 172.21.15.148:0/869405878 >> 172.21.15.148:0/869405878 conn(0x7f097c0f0b70 msgr2=0x7f097c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:43.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.221+0000 7f0983bb3700 1 -- 172.21.15.148:0/869405878 shutdown_connections 2023-12-21T03:44:43.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:43.221+0000 7f0983bb3700 1 -- 172.21.15.148:0/869405878 wait complete. 2023-12-21T03:44:43.223 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2023-12-21T03:44:43.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:44:42.139117+0000 mgr.smithi078.zidbgq (mgr.14162) 165 : cephadm [INF] Deploying daemon mgr.smithi148.tgpyxd on smithi148 2023-12-21T03:44:43.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:42.225745+0000 mgr.smithi078.zidbgq (mgr.14162) 166 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:43.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:43.219784+0000 mon.smithi078 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.148:0/869405878' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:44.544 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-21T03:44:44.544 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-21T03:44:45.317 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:43.873519+0000 mon.smithi078 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:44:45.317 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:43.874794+0000 mon.smithi078 (mon.0) 256 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-21T03:44:45.317 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:43.875582+0000 mon.smithi078 (mon.0) 257 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:44:45.317 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:44:43.876308+0000 mgr.smithi078.zidbgq (mgr.14162) 167 : cephadm [INF] Deploying daemon mon.smithi148 on smithi148 2023-12-21T03:44:45.318 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:44.226151+0000 mgr.smithi078.zidbgq (mgr.14162) 168 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:46.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.099+0000 7f1f550fb700 1 -- 172.21.15.148:0/499056337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f50060220 msgr2=0x7f1f500f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:46.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.099+0000 7f1f550fb700 1 --2- 172.21.15.148:0/499056337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f50060220 0x7f1f500f9150 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f1f40009fb0 tx=0x7f1f40009b30).stop 2023-12-21T03:44:46.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.099+0000 7f1f550fb700 1 -- 172.21.15.148:0/499056337 shutdown_connections 2023-12-21T03:44:46.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.099+0000 7f1f550fb700 1 --2- 172.21.15.148:0/499056337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f50060220 0x7f1f500f9150 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:46.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.099+0000 7f1f550fb700 1 -- 172.21.15.148:0/499056337 >> 172.21.15.148:0/499056337 conn(0x7f1f500f0b70 msgr2=0x7f1f500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:46.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.099+0000 7f1f550fb700 1 -- 172.21.15.148:0/499056337 shutdown_connections 2023-12-21T03:44:46.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.099+0000 7f1f550fb700 1 -- 172.21.15.148:0/499056337 wait complete. 2023-12-21T03:44:46.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.100+0000 7f1f550fb700 1 Processor -- start 2023-12-21T03:44:46.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.100+0000 7f1f550fb700 1 -- start start 2023-12-21T03:44:46.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.100+0000 7f1f550fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f50060220 0x7f1f501066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:46.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.100+0000 7f1f550fb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f50106bd0 con 0x7f1f50060220 2023-12-21T03:44:46.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.101+0000 7f1f4ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f50060220 0x7f1f501066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:46.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.101+0000 7f1f4ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f50060220 0x7f1f501066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.148:35664/0 (socket says 172.21.15.148:35664) 2023-12-21T03:44:46.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.101+0000 7f1f4ed9d700 1 -- 172.21.15.148:0/3771635977 learned_addr learned my addr 172.21.15.148:0/3771635977 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:44:46.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.102+0000 7f1f4ed9d700 1 -- 172.21.15.148:0/3771635977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f40005040 con 0x7f1f50060220 2023-12-21T03:44:46.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.103+0000 7f1f4ed9d700 1 --2- 172.21.15.148:0/3771635977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f50060220 0x7f1f501066e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f1f40009830 tx=0x7f1f4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:46.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.103+0000 7f1f377fe700 1 -- 172.21.15.148:0/3771635977 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f40009320 con 0x7f1f50060220 2023-12-21T03:44:46.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.103+0000 7f1f377fe700 1 -- 172.21.15.148:0/3771635977 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f400136c0 con 0x7f1f50060220 2023-12-21T03:44:46.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.103+0000 7f1f377fe700 1 -- 172.21.15.148:0/3771635977 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f4001ee90 con 0x7f1f50060220 2023-12-21T03:44:46.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.103+0000 7f1f550fb700 1 -- 172.21.15.148:0/3771635977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f50106dd0 con 0x7f1f50060220 2023-12-21T03:44:46.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.103+0000 7f1f550fb700 1 -- 172.21.15.148:0/3771635977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f50107260 con 0x7f1f50060220 2023-12-21T03:44:46.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.104+0000 7f1f377fe700 1 -- 172.21.15.148:0/3771635977 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f4001e420 con 0x7f1f50060220 2023-12-21T03:44:46.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.104+0000 7f1f550fb700 1 -- 172.21.15.148:0/3771635977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f50045ad0 con 0x7f1f50060220 2023-12-21T03:44:46.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.104+0000 7f1f377fe700 1 --2- 172.21.15.148:0/3771635977 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1f3802ff10 0x7f1f380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:46.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.104+0000 7f1f377fe700 1 -- 172.21.15.148:0/3771635977 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f40040ff0 con 0x7f1f50060220 2023-12-21T03:44:46.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.105+0000 7f1f4e59c700 1 --2- 172.21.15.148:0/3771635977 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1f3802ff10 0x7f1f380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:46.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.105+0000 7f1f4e59c700 1 --2- 172.21.15.148:0/3771635977 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1f3802ff10 0x7f1f380323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f1f440096a0 tx=0x7f1f44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:46.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.109+0000 7f1f377fe700 1 -- 172.21.15.148:0/3771635977 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f400095f0 con 0x7f1f50060220 2023-12-21T03:44:46.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.335+0000 7f1f377fe700 1 -- 172.21.15.148:0/3771635977 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f40013ea0 con 0x7f1f50060220 2023-12-21T03:44:46.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:46.347+0000 7f1f550fb700 1 -- 172.21.15.148:0/3771635977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f50108110 con 0x7f1f50060220 2023-12-21T03:44:46.516 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 systemd[1]: Started Ceph mon.smithi148 for 5ef0788c-9fb2-11ee-95a5-87774f69a715. 2023-12-21T03:44:46.516 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.329+0000 7fb08ff4c700 0 mon.smithi148@-1(synchronizing).mds e1 new map 2023-12-21T03:44:46.516 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.329+0000 7fb08ff4c700 0 mon.smithi148@-1(synchronizing).mds e1 print_map 2023-12-21T03:44:46.516 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: e1 2023-12-21T03:44:46.516 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-21T03:44:46.516 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-21T03:44:46.516 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: legacy client fscid: -1 2023-12-21T03:44:46.516 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: No filesystems configured 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 1 mon.smithi148@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 1 mon.smithi148@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 1 mon.smithi148@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 1 mon.smithi148@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 1 mon.smithi148@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 1 mon.smithi148@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 1 mon.smithi148@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 0 mon.smithi148@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 0 mon.smithi148@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 0 mon.smithi148@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 0 mon.smithi148@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2023-12-21T03:44:46.517 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:43.873519+0000 mon.smithi078 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:43.874794+0000 mon.smithi078 (mon.0) 256 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:43.875582+0000 mon.smithi078 (mon.0) 257 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:44:43.876308+0000 mgr.smithi078.zidbgq (mgr.14162) 167 : cephadm [INF] Deploying daemon mon.smithi148 on smithi148 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:44.226151+0000 mgr.smithi078.zidbgq (mgr.14162) 168 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.330+0000 7fb08ff4c700 1 mon.smithi148@-1(synchronizing).paxosservice(auth 1..7) refresh upgraded, format 0 -> 3 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta expand map: {default=false} 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta from 'false' to 'false' 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta expanded map: {default=false} 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta expand map: {default=info} 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta from 'info' to 'info' 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta expanded map: {default=info} 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta expand map: {default=daemon} 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta expanded map: {default=daemon} 2023-12-21T03:44:46.518 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta expand map: {default=debug} 2023-12-21T03:44:46.519 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta from 'debug' to 'debug' 2023-12-21T03:44:46.519 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 20 expand_channel_meta expanded map: {default=debug} 2023-12-21T03:44:46.519 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:46 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:44:46.332+0000 7fb08ff4c700 10 mon.smithi148@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-12-21T03:44:51.372 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-21T03:44:51.372 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":2,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","modified":"2023-12-21T03:44:46.335240Z","created":"2023-12-21T03:38:38.889362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi148","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:3300","nonce":0},{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-12-21T03:44:51.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:51.368+0000 7f1f377fe700 1 -- 172.21.15.148:0/3771635977 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+968 (secure 0 0 0) 0x7f1f4002c050 con 0x7f1f50060220 2023-12-21T03:44:51.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:51.372+0000 7f1f550fb700 1 -- 172.21.15.148:0/3771635977 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1f3802ff10 msgr2=0x7f1f380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:51.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:51.372+0000 7f1f550fb700 1 --2- 172.21.15.148:0/3771635977 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1f3802ff10 0x7f1f380323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f1f440096a0 tx=0x7f1f44006b40).stop 2023-12-21T03:44:51.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:51.372+0000 7f1f550fb700 1 -- 172.21.15.148:0/3771635977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f50060220 msgr2=0x7f1f501066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:51.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:51.372+0000 7f1f550fb700 1 --2- 172.21.15.148:0/3771635977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f50060220 0x7f1f501066e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f1f40009830 tx=0x7f1f4000b040).stop 2023-12-21T03:44:51.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:51.372+0000 7f1f550fb700 1 -- 172.21.15.148:0/3771635977 shutdown_connections 2023-12-21T03:44:51.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:51.372+0000 7f1f550fb700 1 --2- 172.21.15.148:0/3771635977 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1f3802ff10 0x7f1f380323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:51.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:51.372+0000 7f1f550fb700 1 --2- 172.21.15.148:0/3771635977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f50060220 0x7f1f501066e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:51.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:51.372+0000 7f1f550fb700 1 -- 172.21.15.148:0/3771635977 >> 172.21.15.148:0/3771635977 conn(0x7f1f500f0b70 msgr2=0x7f1f500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:51.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:51.372+0000 7f1f550fb700 1 -- 172.21.15.148:0/3771635977 shutdown_connections 2023-12-21T03:44:51.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:44:51.372+0000 7f1f550fb700 1 -- 172.21.15.148:0/3771635977 wait complete. 2023-12-21T03:44:51.374 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 2 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:46.226359+0000 mgr.smithi078.zidbgq (mgr.14162) 169 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:44:46.324263+0000 mgr.smithi078.zidbgq (mgr.14162) 170 : cephadm [INF] Deploying daemon node-exporter.smithi148 on smithi148 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:46.337595+0000 mon.smithi078 (mon.0) 260 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:46.337749+0000 mon.smithi078 (mon.0) 261 : cluster [INF] mon.smithi078 calling monitor election 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:46.350955+0000 mon.smithi078 (mon.0) 262 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:46.351079+0000 mon.smithi078 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.148:0/3771635977' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:47.335400+0000 mon.smithi078 (mon.0) 264 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:48.226705+0000 mgr.smithi078.zidbgq (mgr.14162) 171 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:48.335176+0000 mon.smithi078 (mon.0) 265 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:48.337734+0000 mon.smithi148 (mon.1) 1 : cluster [INF] mon.smithi148 calling monitor election 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:49.029472+0000 mon.smithi078 (mon.0) 266 : audit [DBG] from='mgr.? 172.21.15.148:0/807293696' entity='mgr.smithi148.tgpyxd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi148.tgpyxd/crt"}]: dispatch 2023-12-21T03:44:51.734 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:49.335431+0000 mon.smithi078 (mon.0) 267 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:50.227110+0000 mgr.smithi078.zidbgq (mgr.14162) 172 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:50.335346+0000 mon.smithi078 (mon.0) 268 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:51.335446+0000 mon.smithi078 (mon.0) 269 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:51.364794+0000 mon.smithi078 (mon.0) 270 : cluster [INF] mon.smithi078 is new leader, mons smithi078,smithi148 in quorum (ranks 0,1) 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:51.368540+0000 mon.smithi078 (mon.0) 271 : cluster [DBG] monmap e2: 2 mons at {smithi078=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],smithi148=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0]} 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:51.368590+0000 mon.smithi078 (mon.0) 272 : cluster [DBG] fsmap 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:51.368632+0000 mon.smithi078 (mon.0) 273 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:51.368873+0000 mon.smithi078 (mon.0) 274 : cluster [DBG] mgrmap e17: smithi078.zidbgq(active, since 5m) 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:51.370143+0000 mon.smithi078 (mon.0) 275 : cluster [DBG] Standby manager daemon smithi148.tgpyxd started 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:51.371559+0000 mon.smithi078 (mon.0) 276 : audit [DBG] from='mgr.? 172.21.15.148:0/807293696' entity='mgr.smithi148.tgpyxd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:51.373330+0000 mon.smithi078 (mon.0) 277 : cluster [WRN] Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:51.373348+0000 mon.smithi078 (mon.0) 278 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:51.373365+0000 mon.smithi078 (mon.0) 279 : cluster [WRN] daemon node-exporter.smithi078 on smithi078 is in unknown state 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:51.373707+0000 mon.smithi078 (mon.0) 280 : audit [DBG] from='mgr.? 172.21.15.148:0/807293696' entity='mgr.smithi148.tgpyxd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi148.tgpyxd/key"}]: dispatch 2023-12-21T03:44:51.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:51.374218+0000 mon.smithi078 (mon.0) 281 : audit [DBG] from='mgr.? 172.21.15.148:0/807293696' entity='mgr.smithi148.tgpyxd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-21T03:44:51.736 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-12-21T03:44:51.737 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph config generate-minimal-conf 2023-12-21T03:44:51.772 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:46.226359+0000 mgr.smithi078.zidbgq (mgr.14162) 169 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:51.772 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:44:46.324263+0000 mgr.smithi078.zidbgq (mgr.14162) 170 : cephadm [INF] Deploying daemon node-exporter.smithi148 on smithi148 2023-12-21T03:44:51.772 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:46.337595+0000 mon.smithi078 (mon.0) 260 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2023-12-21T03:44:51.772 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:46.337749+0000 mon.smithi078 (mon.0) 261 : cluster [INF] mon.smithi078 calling monitor election 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:46.350955+0000 mon.smithi078 (mon.0) 262 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:46.351079+0000 mon.smithi078 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.148:0/3771635977' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:47.335400+0000 mon.smithi078 (mon.0) 264 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:48.226705+0000 mgr.smithi078.zidbgq (mgr.14162) 171 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:48.335176+0000 mon.smithi078 (mon.0) 265 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:48.337734+0000 mon.smithi148 (mon.1) 1 : cluster [INF] mon.smithi148 calling monitor election 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:49.029472+0000 mon.smithi078 (mon.0) 266 : audit [DBG] from='mgr.? 172.21.15.148:0/807293696' entity='mgr.smithi148.tgpyxd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi148.tgpyxd/crt"}]: dispatch 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:49.335431+0000 mon.smithi078 (mon.0) 267 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:50.227110+0000 mgr.smithi078.zidbgq (mgr.14162) 172 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:50.335346+0000 mon.smithi078 (mon.0) 268 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:51.335446+0000 mon.smithi078 (mon.0) 269 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:51.364794+0000 mon.smithi078 (mon.0) 270 : cluster [INF] mon.smithi078 is new leader, mons smithi078,smithi148 in quorum (ranks 0,1) 2023-12-21T03:44:51.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:51.368540+0000 mon.smithi078 (mon.0) 271 : cluster [DBG] monmap e2: 2 mons at {smithi078=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],smithi148=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0]} 2023-12-21T03:44:51.774 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:51.368590+0000 mon.smithi078 (mon.0) 272 : cluster [DBG] fsmap 2023-12-21T03:44:51.774 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:51.368632+0000 mon.smithi078 (mon.0) 273 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-12-21T03:44:51.774 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:51.368873+0000 mon.smithi078 (mon.0) 274 : cluster [DBG] mgrmap e17: smithi078.zidbgq(active, since 5m) 2023-12-21T03:44:51.774 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:51.370143+0000 mon.smithi078 (mon.0) 275 : cluster [DBG] Standby manager daemon smithi148.tgpyxd started 2023-12-21T03:44:51.774 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:51.371559+0000 mon.smithi078 (mon.0) 276 : audit [DBG] from='mgr.? 172.21.15.148:0/807293696' entity='mgr.smithi148.tgpyxd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-21T03:44:51.774 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:51.373330+0000 mon.smithi078 (mon.0) 277 : cluster [WRN] Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2023-12-21T03:44:51.774 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:51.373348+0000 mon.smithi078 (mon.0) 278 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-12-21T03:44:51.774 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:51.373365+0000 mon.smithi078 (mon.0) 279 : cluster [WRN] daemon node-exporter.smithi078 on smithi078 is in unknown state 2023-12-21T03:44:51.774 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:51.373707+0000 mon.smithi078 (mon.0) 280 : audit [DBG] from='mgr.? 172.21.15.148:0/807293696' entity='mgr.smithi148.tgpyxd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi148.tgpyxd/key"}]: dispatch 2023-12-21T03:44:51.774 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:51.374218+0000 mon.smithi078 (mon.0) 281 : audit [DBG] from='mgr.? 172.21.15.148:0/807293696' entity='mgr.smithi148.tgpyxd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-21T03:44:52.729 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:51.429821+0000 mon.smithi078 (mon.0) 282 : cluster [DBG] mgrmap e18: smithi078.zidbgq(active, since 5m), standbys: smithi148.tgpyxd 2023-12-21T03:44:52.730 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:51.429978+0000 mon.smithi078 (mon.0) 283 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mgr metadata", "who": "smithi148.tgpyxd", "id": "smithi148.tgpyxd"}]: dispatch 2023-12-21T03:44:52.730 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:52.335553+0000 mon.smithi078 (mon.0) 284 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:52.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:52 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:51.429821+0000 mon.smithi078 (mon.0) 282 : cluster [DBG] mgrmap e18: smithi078.zidbgq(active, since 5m), standbys: smithi148.tgpyxd 2023-12-21T03:44:52.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:52 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:51.429978+0000 mon.smithi078 (mon.0) 283 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mgr metadata", "who": "smithi148.tgpyxd", "id": "smithi148.tgpyxd"}]: dispatch 2023-12-21T03:44:52.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:52 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:52.335553+0000 mon.smithi078 (mon.0) 284 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mon metadata", "id": "smithi148"}]: dispatch 2023-12-21T03:44:52.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.943+0000 7f4171e6b700 1 -- 172.21.15.78:0/4096339009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0f5410 msgr2=0x7f416c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:52.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.943+0000 7f4171e6b700 1 --2- 172.21.15.78:0/4096339009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0f5410 0x7f416c0f5830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f415c009fb0 tx=0x7f415c009b30).stop 2023-12-21T03:44:52.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.943+0000 7f4171e6b700 1 -- 172.21.15.78:0/4096339009 shutdown_connections 2023-12-21T03:44:52.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.943+0000 7f4171e6b700 1 --2- 172.21.15.78:0/4096339009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0f5410 0x7f416c0f5830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:52.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.943+0000 7f4171e6b700 1 -- 172.21.15.78:0/4096339009 >> 172.21.15.78:0/4096339009 conn(0x7f416c0f0b70 msgr2=0x7f416c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:52.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.943+0000 7f4171e6b700 1 -- 172.21.15.78:0/4096339009 shutdown_connections 2023-12-21T03:44:52.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.943+0000 7f4171e6b700 1 -- 172.21.15.78:0/4096339009 wait complete. 2023-12-21T03:44:52.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f4171e6b700 1 Processor -- start 2023-12-21T03:44:52.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f4171e6b700 1 -- start start 2023-12-21T03:44:52.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f4171e6b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0f5410 0x7f416c106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:52.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f4171e6b700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f416c106d10 0x7f416c107c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:52.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f4171e6b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c107270 con 0x7f416c0f5410 2023-12-21T03:44:52.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f4171e6b700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c1073b0 con 0x7f416c106d10 2023-12-21T03:44:52.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f416b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0f5410 0x7f416c106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:52.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f416b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0f5410 0x7f416c106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35768/0 (socket says 172.21.15.78:35768) 2023-12-21T03:44:52.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f416b7fe700 1 -- 172.21.15.78:0/3635919224 learned_addr learned my addr 172.21.15.78:0/3635919224 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:44:52.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f416b7fe700 1 -- 172.21.15.78:0/3635919224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f416c106d10 msgr2=0x7f416c107c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:52.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f416b7fe700 1 --2- 172.21.15.78:0/3635919224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f416c106d10 0x7f416c107c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:44:52.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f416b7fe700 1 -- 172.21.15.78:0/3635919224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c005040 con 0x7f416c0f5410 2023-12-21T03:44:52.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.944+0000 7f416b7fe700 1 --2- 172.21.15.78:0/3635919224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0f5410 0x7f416c106820 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f415c000c00 tx=0x7f415c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:52.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.945+0000 7f4153fff700 1 -- 172.21.15.78:0/3635919224 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f415c019070 con 0x7f416c0f5410 2023-12-21T03:44:52.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.945+0000 7f4171e6b700 1 -- 172.21.15.78:0/3635919224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c1081b0 con 0x7f416c0f5410 2023-12-21T03:44:52.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.945+0000 7f4153fff700 1 -- 172.21.15.78:0/3635919224 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f415c007ba0 con 0x7f416c0f5410 2023-12-21T03:44:52.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.945+0000 7f4153fff700 1 -- 172.21.15.78:0/3635919224 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f415c01d3c0 con 0x7f416c0f5410 2023-12-21T03:44:52.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.945+0000 7f4171e6b700 1 -- 172.21.15.78:0/3635919224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c108710 con 0x7f416c0f5410 2023-12-21T03:44:52.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.946+0000 7f4171e6b700 1 -- 172.21.15.78:0/3635919224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416c045ad0 con 0x7f416c0f5410 2023-12-21T03:44:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.946+0000 7f4153fff700 1 -- 172.21.15.78:0/3635919224 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f415c025070 con 0x7f416c0f5410 2023-12-21T03:44:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.946+0000 7f4153fff700 1 --2- 172.21.15.78:0/3635919224 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f415405c3b0 0x7f415405e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.946+0000 7f4153fff700 1 -- 172.21.15.78:0/3635919224 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f415c0761c0 con 0x7f416c0f5410 2023-12-21T03:44:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.946+0000 7f416affd700 1 --2- 172.21.15.78:0/3635919224 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f415405c3b0 0x7f415405e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.947+0000 7f416affd700 1 --2- 172.21.15.78:0/3635919224 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f415405c3b0 0x7f415405e870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f41600095b0 tx=0x7f4160006ae0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:52.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:52.950+0000 7f4153fff700 1 -- 172.21.15.78:0/3635919224 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f415c049e90 con 0x7f416c0f5410 2023-12-21T03:44:53.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.100+0000 7f4171e6b700 1 -- 172.21.15.78:0/3635919224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f416c0008d0 con 0x7f416c0f5410 2023-12-21T03:44:53.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.101+0000 7f4153fff700 1 -- 172.21.15.78:0/3635919224 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v12) v1 ==== 76+0+225 (secure 0 0 0) 0x7f415c04d4b0 con 0x7f416c0f5410 2023-12-21T03:44:53.102 INFO:teuthology.orchestra.run.smithi078.stdout:# minimal ceph.conf for 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:44:53.102 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2023-12-21T03:44:53.103 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = 5ef0788c-9fb2-11ee-95a5-87774f69a715 2023-12-21T03:44:53.103 INFO:teuthology.orchestra.run.smithi078.stdout: mon_host = [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-12-21T03:44:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.102+0000 7f4171e6b700 1 -- 172.21.15.78:0/3635919224 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f415405c3b0 msgr2=0x7f415405e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.102+0000 7f4171e6b700 1 --2- 172.21.15.78:0/3635919224 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f415405c3b0 0x7f415405e870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f41600095b0 tx=0x7f4160006ae0).stop 2023-12-21T03:44:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.102+0000 7f4171e6b700 1 -- 172.21.15.78:0/3635919224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0f5410 msgr2=0x7f416c106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.102+0000 7f4171e6b700 1 --2- 172.21.15.78:0/3635919224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0f5410 0x7f416c106820 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f415c000c00 tx=0x7f415c00b040).stop 2023-12-21T03:44:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.102+0000 7f4171e6b700 1 -- 172.21.15.78:0/3635919224 shutdown_connections 2023-12-21T03:44:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.102+0000 7f4171e6b700 1 --2- 172.21.15.78:0/3635919224 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f415405c3b0 0x7f415405e870 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.102+0000 7f4171e6b700 1 --2- 172.21.15.78:0/3635919224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0f5410 0x7f416c106820 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.102+0000 7f4171e6b700 1 --2- 172.21.15.78:0/3635919224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f416c106d10 0x7f416c107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:44:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.102+0000 7f4171e6b700 1 -- 172.21.15.78:0/3635919224 >> 172.21.15.78:0/3635919224 conn(0x7f416c0f0b70 msgr2=0x7f416c05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.103+0000 7f4171e6b700 1 -- 172.21.15.78:0/3635919224 shutdown_connections 2023-12-21T03:44:53.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:53.103+0000 7f4171e6b700 1 -- 172.21.15.78:0/3635919224 wait complete. 2023-12-21T03:44:53.482 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-12-21T03:44:53.482 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:44:53.482 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.conf 2023-12-21T03:44:53.511 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:44:53.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-21T03:44:53.578 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:44:53.578 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.conf 2023-12-21T03:44:53.617 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:44:53.617 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-21T03:44:53.685 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:52.227378+0000 mgr.smithi078.zidbgq (mgr.14162) 173 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:53.685 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:53.102189+0000 mon.smithi078 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.78:0/3635919224' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:44:53.689 INFO:tasks.cephadm:Deploying OSDs... 2023-12-21T03:44:53.689 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:44:53.689 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2023-12-21T03:44:53.706 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-21T03:44:53.706 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2023-12-21T03:44:53.763 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-21T03:44:53.764 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-21T03:44:53.764 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 392362 Links: 1 2023-12-21T03:44:53.764 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-21T03:44:53.764 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-12-21T03:44:53.764 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-12-21 03:40:07.635948945 +0000 2023-12-21T03:44:53.764 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-12-21 03:33:52.514020621 +0000 2023-12-21T03:44:53.764 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-12-21 03:33:52.514020621 +0000 2023-12-21T03:44:53.764 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-12-21T03:44:53.764 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-21T03:44:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:52.227378+0000 mgr.smithi078.zidbgq (mgr.14162) 173 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:53.102189+0000 mon.smithi078 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.78:0/3635919224' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:44:53.829 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-12-21T03:44:53.830 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-12-21T03:44:53.830 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000221001 s, 2.3 MB/s 2023-12-21T03:44:53.831 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-21T03:44:53.890 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2023-12-21T03:44:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-21T03:44:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-21T03:44:53.949 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 395524 Links: 1 2023-12-21T03:44:53.949 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-21T03:44:53.949 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-12-21T03:44:53.950 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-12-21 03:40:07.635948945 +0000 2023-12-21T03:44:53.950 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-12-21 03:33:53.023012095 +0000 2023-12-21T03:44:53.950 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-12-21 03:33:53.023012095 +0000 2023-12-21T03:44:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-12-21T03:44:53.950 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-21T03:44:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-12-21T03:44:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-12-21T03:44:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000213873 s, 2.4 MB/s 2023-12-21T03:44:54.017 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-21T03:44:54.076 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2023-12-21T03:44:54.134 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-21T03:44:54.134 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-21T03:44:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 395614 Links: 1 2023-12-21T03:44:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-21T03:44:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-12-21T03:44:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-12-21 03:40:07.635948945 +0000 2023-12-21T03:44:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-12-21 03:33:53.539003452 +0000 2023-12-21T03:44:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-12-21 03:33:53.539003452 +0000 2023-12-21T03:44:54.135 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-12-21T03:44:54.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-21T03:44:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-12-21T03:44:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-12-21T03:44:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000188332 s, 2.7 MB/s 2023-12-21T03:44:54.201 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-21T03:44:54.261 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2023-12-21T03:44:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-21T03:44:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-21T03:44:54.319 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 395952 Links: 1 2023-12-21T03:44:54.319 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-21T03:44:54.320 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-12-21T03:44:54.320 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-12-21 03:40:07.635948945 +0000 2023-12-21T03:44:54.320 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-12-21 03:33:54.070994540 +0000 2023-12-21T03:44:54.320 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-12-21 03:33:54.070994540 +0000 2023-12-21T03:44:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-12-21T03:44:54.321 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-21T03:44:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-12-21T03:44:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-12-21T03:44:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000231385 s, 2.2 MB/s 2023-12-21T03:44:54.386 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-21T03:44:54.445 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:44:54.445 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/scratch_devs of=/dev/stdout 2023-12-21T03:44:54.468 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-21T03:44:54.468 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 2023-12-21T03:44:54.529 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-21T03:44:54.529 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-21T03:44:54.529 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 395470 Links: 1 2023-12-21T03:44:54.529 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-21T03:44:54.529 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2023-12-21T03:44:54.529 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-12-21 03:44:46.764475497 +0000 2023-12-21T03:44:54.529 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-12-21 03:33:52.979320268 +0000 2023-12-21T03:44:54.530 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-12-21 03:33:52.979320268 +0000 2023-12-21T03:44:54.530 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2023-12-21T03:44:54.530 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-21T03:44:54.598 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-12-21T03:44:54.598 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-12-21T03:44:54.598 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000227215 s, 2.3 MB/s 2023-12-21T03:44:54.599 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-21T03:44:54.661 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 2023-12-21T03:44:54.721 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-21T03:44:54.722 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-21T03:44:54.722 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 394472 Links: 1 2023-12-21T03:44:54.722 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-21T03:44:54.722 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2023-12-21T03:44:54.722 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-12-21 03:44:46.764475497 +0000 2023-12-21T03:44:54.722 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-12-21 03:33:53.495307920 +0000 2023-12-21T03:44:54.722 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-12-21 03:33:53.495307920 +0000 2023-12-21T03:44:54.722 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2023-12-21T03:44:54.723 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-21T03:44:54.790 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-12-21T03:44:54.790 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-12-21T03:44:54.790 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000221307 s, 2.3 MB/s 2023-12-21T03:44:54.791 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-21T03:44:54.852 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 2023-12-21T03:44:54.912 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-21T03:44:54.913 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-21T03:44:54.913 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 393897 Links: 1 2023-12-21T03:44:54.913 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-21T03:44:54.913 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2023-12-21T03:44:54.913 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-12-21 03:44:46.764475497 +0000 2023-12-21T03:44:54.914 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-12-21 03:33:54.019295381 +0000 2023-12-21T03:44:54.914 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-12-21 03:33:54.019295381 +0000 2023-12-21T03:44:54.914 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2023-12-21T03:44:54.914 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-21T03:44:54.980 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-12-21T03:44:54.980 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-12-21T03:44:54.980 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000285084 s, 1.8 MB/s 2023-12-21T03:44:54.981 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-21T03:44:55.041 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 2023-12-21T03:44:55.098 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-21T03:44:55.099 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-21T03:44:55.099 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 394617 Links: 1 2023-12-21T03:44:55.099 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-21T03:44:55.099 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2023-12-21T03:44:55.099 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-12-21 03:44:46.764475497 +0000 2023-12-21T03:44:55.099 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-12-21 03:33:54.570282196 +0000 2023-12-21T03:44:55.099 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-12-21 03:33:54.570282196 +0000 2023-12-21T03:44:55.099 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2023-12-21T03:44:55.099 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-21T03:44:55.164 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-12-21T03:44:55.164 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-12-21T03:44:55.164 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.00020207 s, 2.5 MB/s 2023-12-21T03:44:55.165 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-21T03:44:55.224 INFO:tasks.cephadm:Deploying osd.0 on smithi078 with /dev/vg_nvme/lv_4... 2023-12-21T03:44:55.224 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-21T03:44:55.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:54.227658+0000 mgr.smithi078.zidbgq (mgr.14162) 174 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:54.227658+0000 mgr.smithi078.zidbgq (mgr.14162) 174 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:56.733 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-21T03:44:56.733 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-21T03:44:56.733 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-12-21T03:44:56.733 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-12-21T03:44:56.733 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.015228 s, 689 MB/s 2023-12-21T03:44:56.733 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-12-21T03:44:57.109 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_4 2023-12-21T03:44:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:56.227913+0000 mgr.smithi078.zidbgq (mgr.14162) 175 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:57.812 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:56.227913+0000 mgr.smithi078.zidbgq (mgr.14162) 175 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.358+0000 7fe9b8137700 1 -- 172.21.15.78:0/1060942609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b00f5fe0 msgr2=0x7fe9b00f6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.358+0000 7fe9b8137700 1 --2- 172.21.15.78:0/1060942609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b00f5fe0 0x7fe9b00f6460 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fe9a0004660 tx=0x7fe9a0009b30).stop 2023-12-21T03:44:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 -- 172.21.15.78:0/1060942609 shutdown_connections 2023-12-21T03:44:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 --2- 172.21.15.78:0/1060942609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b00f5fe0 0x7fe9b00f6460 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 --2- 172.21.15.78:0/1060942609 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9b00f4de0 0x7fe9b00f5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:44:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 -- 172.21.15.78:0/1060942609 >> 172.21.15.78:0/1060942609 conn(0x7fe9b00f0540 msgr2=0x7fe9b00f2960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:44:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 -- 172.21.15.78:0/1060942609 shutdown_connections 2023-12-21T03:44:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 -- 172.21.15.78:0/1060942609 wait complete. 2023-12-21T03:44:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 Processor -- start 2023-12-21T03:44:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 -- start start 2023-12-21T03:44:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b00f4de0 0x7fe9b0109cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9b00f5fe0 0x7fe9b010c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00fc050 con 0x7fe9b00f4de0 2023-12-21T03:44:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b8137700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00fc190 con 0x7fe9b00f5fe0 2023-12-21T03:44:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b5ed3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b00f4de0 0x7fe9b0109cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b5ed3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b00f4de0 0x7fe9b0109cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47494/0 (socket says 172.21.15.78:47494) 2023-12-21T03:44:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.359+0000 7fe9b5ed3700 1 -- 172.21.15.78:0/4131361557 learned_addr learned my addr 172.21.15.78:0/4131361557 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:44:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.360+0000 7fe9b56d2700 1 --2- 172.21.15.78:0/4131361557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9b00f5fe0 0x7fe9b010c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.360+0000 7fe9b5ed3700 1 -- 172.21.15.78:0/4131361557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9b00f5fe0 msgr2=0x7fe9b010c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:44:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.360+0000 7fe9b5ed3700 1 --2- 172.21.15.78:0/4131361557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9b00f5fe0 0x7fe9b010c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:44:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.360+0000 7fe9b5ed3700 1 -- 172.21.15.78:0/4131361557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a0005040 con 0x7fe9b00f4de0 2023-12-21T03:44:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.360+0000 7fe9b5ed3700 1 --2- 172.21.15.78:0/4131361557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b00f4de0 0x7fe9b0109cd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fe9ac00b410 tx=0x7fe9ac006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.360+0000 7fe9a67fc700 1 -- 172.21.15.78:0/4131361557 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9ac0125f0 con 0x7fe9b00f4de0 2023-12-21T03:44:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.360+0000 7fe9a67fc700 1 -- 172.21.15.78:0/4131361557 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9ac00be80 con 0x7fe9b00f4de0 2023-12-21T03:44:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.360+0000 7fe9b8137700 1 -- 172.21.15.78:0/4131361557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b010c7e0 con 0x7fe9b00f4de0 2023-12-21T03:44:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.360+0000 7fe9a67fc700 1 -- 172.21.15.78:0/4131361557 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9ac002ac0 con 0x7fe9b00f4de0 2023-12-21T03:44:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.360+0000 7fe9b8137700 1 -- 172.21.15.78:0/4131361557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b010cda0 con 0x7fe9b00f4de0 2023-12-21T03:44:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.361+0000 7fe9b8137700 1 -- 172.21.15.78:0/4131361557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b0045ad0 con 0x7fe9b00f4de0 2023-12-21T03:44:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.361+0000 7fe9a67fc700 1 -- 172.21.15.78:0/4131361557 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe9ac00f030 con 0x7fe9b00f4de0 2023-12-21T03:44:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.362+0000 7fe9a67fc700 1 --2- 172.21.15.78:0/4131361557 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe99c05c2b0 0x7fe99c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:44:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.362+0000 7fe9a67fc700 1 -- 172.21.15.78:0/4131361557 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9ac075390 con 0x7fe9b00f4de0 2023-12-21T03:44:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.362+0000 7fe9b56d2700 1 --2- 172.21.15.78:0/4131361557 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe99c05c2b0 0x7fe99c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:44:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.362+0000 7fe9b56d2700 1 --2- 172.21.15.78:0/4131361557 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe99c05c2b0 0x7fe99c05e770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe9a0013580 tx=0x7fe9a0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:44:58.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.366+0000 7fe9a67fc700 1 -- 172.21.15.78:0/4131361557 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9ac049000 con 0x7fe9b00f4de0 2023-12-21T03:44:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:44:58.540+0000 7fe9b8137700 1 -- 172.21.15.78:0/4131361557 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9b00fdfb0 con 0x7fe99c05c2b0 2023-12-21T03:44:59.527 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:44:58.228298+0000 mgr.smithi078.zidbgq (mgr.14162) 176 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:59.527 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:58.543244+0000 mon.smithi078 (mon.0) 286 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:44:59.527 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:58.544644+0000 mon.smithi078 (mon.0) 287 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:44:59.527 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:44:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:58.545099+0000 mon.smithi078 (mon.0) 288 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:44:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:44:58.228298+0000 mgr.smithi078.zidbgq (mgr.14162) 176 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:44:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:58.543244+0000 mon.smithi078 (mon.0) 286 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:44:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:58.544644+0000 mon.smithi078 (mon.0) 287 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:44:59.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:44:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:58.545099+0000 mon.smithi078 (mon.0) 288 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:00.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:44:58.542514+0000 mgr.smithi078.zidbgq (mgr.14162) 177 : audit [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:45:00.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:44:58.544437+0000 mgr.smithi078.zidbgq (mgr.14162) 178 : cephadm [INF] Applying service osd.None on host smithi078... 2023-12-21T03:45:00.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:00.200776+0000 mon.smithi078 (mon.0) 289 : audit [INF] from='client.? 172.21.15.78:0/2974897572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd284657-10ac-4999-b6e0-ecb000aa6e81"}]: dispatch 2023-12-21T03:45:00.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:00.206745+0000 mon.smithi078 (mon.0) 290 : audit [INF] from='client.? 172.21.15.78:0/2974897572' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd284657-10ac-4999-b6e0-ecb000aa6e81"}]': finished 2023-12-21T03:45:00.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:00.206789+0000 mon.smithi078 (mon.0) 291 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-12-21T03:45:00.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:00.206901+0000 mon.smithi078 (mon.0) 292 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-21T03:45:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:44:58.542514+0000 mgr.smithi078.zidbgq (mgr.14162) 177 : audit [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:45:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:44:58.544437+0000 mgr.smithi078.zidbgq (mgr.14162) 178 : cephadm [INF] Applying service osd.None on host smithi078... 2023-12-21T03:45:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:00.200776+0000 mon.smithi078 (mon.0) 289 : audit [INF] from='client.? 172.21.15.78:0/2974897572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd284657-10ac-4999-b6e0-ecb000aa6e81"}]: dispatch 2023-12-21T03:45:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:00.206745+0000 mon.smithi078 (mon.0) 290 : audit [INF] from='client.? 172.21.15.78:0/2974897572' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd284657-10ac-4999-b6e0-ecb000aa6e81"}]': finished 2023-12-21T03:45:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:00.206789+0000 mon.smithi078 (mon.0) 291 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-12-21T03:45:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:00.206901+0000 mon.smithi078 (mon.0) 292 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-21T03:45:01.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:00.228557+0000 mgr.smithi078.zidbgq (mgr.14162) 179 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:01.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:00.618196+0000 mon.smithi078 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.78:0/1820425154' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:45:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:00.228557+0000 mgr.smithi078.zidbgq (mgr.14162) 179 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:00.618196+0000 mon.smithi078 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.78:0/1820425154' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:45:03.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:02.228931+0000 mgr.smithi078.zidbgq (mgr.14162) 180 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:02.228931+0000 mgr.smithi078.zidbgq (mgr.14162) 180 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:05.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:04.229197+0000 mgr.smithi078.zidbgq (mgr.14162) 181 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:04.229197+0000 mgr.smithi078.zidbgq (mgr.14162) 181 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:06.702 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:06 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:05.989632+0000 mon.smithi078 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-21T03:45:06.703 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:06 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:05.990234+0000 mon.smithi078 (mon.0) 295 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:06.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:06 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:05.989632+0000 mon.smithi078 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-21T03:45:06.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:06 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:05.990234+0000 mon.smithi078 (mon.0) 295 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:07.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:45:05.990658+0000 mgr.smithi078.zidbgq (mgr.14162) 182 : cephadm [INF] Deploying daemon osd.0 on smithi078 2023-12-21T03:45:07.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:06.229433+0000 mgr.smithi078.zidbgq (mgr.14162) 183 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:07.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:45:05.990658+0000 mgr.smithi078.zidbgq (mgr.14162) 182 : cephadm [INF] Deploying daemon osd.0 on smithi078 2023-12-21T03:45:07.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:06.229433+0000 mgr.smithi078.zidbgq (mgr.14162) 183 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:08.482 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 0 on host 'smithi078' 2023-12-21T03:45:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.479+0000 7fe9a67fc700 1 -- 172.21.15.78:0/4131361557 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe9a0019040 con 0x7fe99c05c2b0 2023-12-21T03:45:08.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.482+0000 7fe9b8137700 1 -- 172.21.15.78:0/4131361557 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe99c05c2b0 msgr2=0x7fe99c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:08.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.482+0000 7fe9b8137700 1 --2- 172.21.15.78:0/4131361557 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe99c05c2b0 0x7fe99c05e770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe9a0013580 tx=0x7fe9a0009b30).stop 2023-12-21T03:45:08.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.482+0000 7fe9b8137700 1 -- 172.21.15.78:0/4131361557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b00f4de0 msgr2=0x7fe9b0109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:08.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.482+0000 7fe9b8137700 1 --2- 172.21.15.78:0/4131361557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b00f4de0 0x7fe9b0109cd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fe9ac00b410 tx=0x7fe9ac006ef0).stop 2023-12-21T03:45:08.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.482+0000 7fe9b8137700 1 -- 172.21.15.78:0/4131361557 shutdown_connections 2023-12-21T03:45:08.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.482+0000 7fe9b8137700 1 --2- 172.21.15.78:0/4131361557 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe99c05c2b0 0x7fe99c05e770 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:08.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.482+0000 7fe9b8137700 1 --2- 172.21.15.78:0/4131361557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b00f4de0 0x7fe9b0109cd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:08.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.482+0000 7fe9b8137700 1 --2- 172.21.15.78:0/4131361557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9b00f5fe0 0x7fe9b010c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:08.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.482+0000 7fe9b8137700 1 -- 172.21.15.78:0/4131361557 >> 172.21.15.78:0/4131361557 conn(0x7fe9b00f0540 msgr2=0x7fe9b00f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:45:08.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.482+0000 7fe9b8137700 1 -- 172.21.15.78:0/4131361557 shutdown_connections 2023-12-21T03:45:08.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:08.482+0000 7fe9b8137700 1 -- 172.21.15.78:0/4131361557 wait complete. 2023-12-21T03:45:08.866 DEBUG:teuthology.orchestra.run.smithi078:osd.0> sudo journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.0.service 2023-12-21T03:45:08.868 INFO:tasks.cephadm:Deploying osd.1 on smithi078 with /dev/vg_nvme/lv_3... 2023-12-21T03:45:08.868 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-21T03:45:08.899 INFO:journalctl@ceph.osd.0.smithi078.stdout:-- Logs begin at Thu 2023-12-21 03:26:39 UTC. -- 2023-12-21T03:45:09.201 INFO:journalctl@ceph.osd.0.smithi078.stdout:Dec 21 03:45:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.0[120755]: debug 2023-12-21T03:45:08.933+0000 7fefbf49a080 -1 Falling back to public interface 2023-12-21T03:45:09.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:08.229696+0000 mgr.smithi078.zidbgq (mgr.14162) 184 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:09.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:08.476571+0000 mon.smithi078 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:09.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:08.479936+0000 mon.smithi078 (mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:09.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:08.229696+0000 mgr.smithi078.zidbgq (mgr.14162) 184 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:09.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:08.476571+0000 mon.smithi078 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:09.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:08.479936+0000 mon.smithi078 (mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:10.314 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-21T03:45:10.314 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-21T03:45:10.314 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-12-21T03:45:10.314 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-12-21T03:45:10.314 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0136965 s, 766 MB/s 2023-12-21T03:45:10.314 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-12-21T03:45:10.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_3 2023-12-21T03:45:11.727 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:10.229955+0000 mgr.smithi078.zidbgq (mgr.14162) 185 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:10.229955+0000 mgr.smithi078.zidbgq (mgr.14162) 185 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.919+0000 7f846b947700 1 -- 172.21.15.78:0/1664128133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640f5bb0 msgr2=0x7f84640f5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.919+0000 7f846b947700 1 --2- 172.21.15.78:0/1664128133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640f5bb0 0x7f84640f5fd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f8454004660 tx=0x7f8454009b30).stop 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.919+0000 7f846b947700 1 -- 172.21.15.78:0/1664128133 shutdown_connections 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.919+0000 7f846b947700 1 --2- 172.21.15.78:0/1664128133 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84640f6550 0x7f84640f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.919+0000 7f846b947700 1 --2- 172.21.15.78:0/1664128133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640f5bb0 0x7f84640f5fd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.919+0000 7f846b947700 1 -- 172.21.15.78:0/1664128133 >> 172.21.15.78:0/1664128133 conn(0x7f84640f04c0 msgr2=0x7f84640f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.919+0000 7f846b947700 1 -- 172.21.15.78:0/1664128133 shutdown_connections 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.919+0000 7f846b947700 1 -- 172.21.15.78:0/1664128133 wait complete. 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f846b947700 1 Processor -- start 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f846b947700 1 -- start start 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f846b947700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640f5bb0 0x7f8464109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:45:11.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f846b947700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84640f6550 0x7f846410c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f846b947700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84640f56e0 con 0x7f84640f5bb0 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f846b947700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84640f5820 con 0x7f84640f6550 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f84696e3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640f5bb0 0x7f8464109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f84696e3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640f5bb0 0x7f8464109c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60126/0 (socket says 172.21.15.78:60126) 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f84696e3700 1 -- 172.21.15.78:0/3287388580 learned_addr learned my addr 172.21.15.78:0/3287388580 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f8468ee2700 1 --2- 172.21.15.78:0/3287388580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84640f6550 0x7f846410c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f84696e3700 1 -- 172.21.15.78:0/3287388580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84640f6550 msgr2=0x7f846410c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f84696e3700 1 --2- 172.21.15.78:0/3287388580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84640f6550 0x7f846410c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.920+0000 7f84696e3700 1 -- 172.21.15.78:0/3287388580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8454005040 con 0x7f84640f5bb0 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.921+0000 7f84696e3700 1 --2- 172.21.15.78:0/3287388580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640f5bb0 0x7f8464109c70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f8454007a60 tx=0x7f8454009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.921+0000 7f8459ffb700 1 -- 172.21.15.78:0/3287388580 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8454018070 con 0x7f84640f5bb0 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.921+0000 7f8459ffb700 1 -- 172.21.15.78:0/3287388580 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8454013410 con 0x7f84640f5bb0 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.921+0000 7f846b947700 1 -- 172.21.15.78:0/3287388580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846410c720 con 0x7f84640f5bb0 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.921+0000 7f8459ffb700 1 -- 172.21.15.78:0/3287388580 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f845401c620 con 0x7f84640f5bb0 2023-12-21T03:45:11.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.921+0000 7f846b947700 1 -- 172.21.15.78:0/3287388580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846410cc50 con 0x7f84640f5bb0 2023-12-21T03:45:11.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.922+0000 7f846b947700 1 -- 172.21.15.78:0/3287388580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84640f4150 con 0x7f84640f5bb0 2023-12-21T03:45:11.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.922+0000 7f8459ffb700 1 -- 172.21.15.78:0/3287388580 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f845401d420 con 0x7f84640f5bb0 2023-12-21T03:45:11.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.922+0000 7f8459ffb700 1 --2- 172.21.15.78:0/3287388580 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f845005c2a0 0x7f845005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:45:11.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.923+0000 7f8459ffb700 1 -- 172.21.15.78:0/3287388580 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1385+0+0 (secure 0 0 0) 0x7f8454076ce0 con 0x7f84640f5bb0 2023-12-21T03:45:11.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.923+0000 7f8468ee2700 1 --2- 172.21.15.78:0/3287388580 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f845005c2a0 0x7f845005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:45:11.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.923+0000 7f8468ee2700 1 --2- 172.21.15.78:0/3287388580 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f845005c2a0 0x7f845005e760 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f8460004a70 tx=0x7f846000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:45:11.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:11.926+0000 7f8459ffb700 1 -- 172.21.15.78:0/3287388580 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f845404a9c0 con 0x7f84640f5bb0 2023-12-21T03:45:12.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:12.101+0000 7f846b947700 1 -- 172.21.15.78:0/3287388580 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f84640fde50 con 0x7f845005c2a0 2023-12-21T03:45:12.729 INFO:journalctl@ceph.osd.0.smithi078.stdout:Dec 21 03:45:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.0[120755]: debug 2023-12-21T03:45:12.425+0000 7fefbf49a080 -1 osd.0 0 log_to_monitors {default=true} 2023-12-21T03:45:12.730 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:12.103812+0000 mon.smithi078 (mon.0) 298 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:45:12.730 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:12.104962+0000 mon.smithi078 (mon.0) 299 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:45:12.730 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:12.105437+0000 mon.smithi078 (mon.0) 300 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:12.730 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:12.430206+0000 mon.smithi078 (mon.0) 301 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-21T03:45:12.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:12 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:12.103812+0000 mon.smithi078 (mon.0) 298 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:45:12.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:12 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:12.104962+0000 mon.smithi078 (mon.0) 299 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:45:12.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:12 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:12.105437+0000 mon.smithi078 (mon.0) 300 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:12.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:12 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:12.430206+0000 mon.smithi078 (mon.0) 301 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-21T03:45:13.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:12.103151+0000 mgr.smithi078.zidbgq (mgr.14162) 186 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:45:13.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:45:12.104767+0000 mgr.smithi078.zidbgq (mgr.14162) 187 : cephadm [INF] Applying service osd.None on host smithi078... 2023-12-21T03:45:13.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:12.230242+0000 mgr.smithi078.zidbgq (mgr.14162) 188 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:13.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:12.482569+0000 mon.smithi078 (mon.0) 302 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-21T03:45:13.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:12.482609+0000 mon.smithi078 (mon.0) 303 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2023-12-21T03:45:13.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:12.482980+0000 mon.smithi078 (mon.0) 304 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-21T03:45:13.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:12.483271+0000 mon.smithi078 (mon.0) 305 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-12-21T03:45:13.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:12.103151+0000 mgr.smithi078.zidbgq (mgr.14162) 186 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:45:13.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:45:12.104767+0000 mgr.smithi078.zidbgq (mgr.14162) 187 : cephadm [INF] Applying service osd.None on host smithi078... 2023-12-21T03:45:13.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:12.230242+0000 mgr.smithi078.zidbgq (mgr.14162) 188 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:13.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:12.482569+0000 mon.smithi078 (mon.0) 302 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-21T03:45:13.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:12.482609+0000 mon.smithi078 (mon.0) 303 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2023-12-21T03:45:13.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:12.482980+0000 mon.smithi078 (mon.0) 304 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-21T03:45:13.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:12.483271+0000 mon.smithi078 (mon.0) 305 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-12-21T03:45:13.818 INFO:journalctl@ceph.osd.0.smithi078.stdout:Dec 21 03:45:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.0[120755]: debug 2023-12-21T03:45:13.486+0000 7fefb5a0b700 -1 osd.0 0 waiting for initial osdmap 2023-12-21T03:45:13.818 INFO:journalctl@ceph.osd.0.smithi078.stdout:Dec 21 03:45:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.0[120755]: debug 2023-12-21T03:45:13.489+0000 7fefb0183700 -1 osd.0 8 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2023-12-21T03:45:14.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:14 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:13.483789+0000 mon.smithi078 (mon.0) 306 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-12-21T03:45:14.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:14 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:13.483838+0000 mon.smithi078 (mon.0) 307 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2023-12-21T03:45:14.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:14 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:13.483924+0000 mon.smithi078 (mon.0) 308 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-21T03:45:14.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:14 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:13.488256+0000 mon.smithi078 (mon.0) 309 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-21T03:45:14.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:14 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:13.681741+0000 mon.smithi078 (mon.0) 310 : audit [INF] from='client.? 172.21.15.78:0/2375551490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c36ef007-5c1b-4aa8-bee6-db97e9852730"}]: dispatch 2023-12-21T03:45:14.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:14 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:13.688555+0000 mon.smithi078 (mon.0) 311 : cluster [INF] osd.0 [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166] boot 2023-12-21T03:45:14.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:14 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:13.688599+0000 mon.smithi078 (mon.0) 312 : audit [INF] from='client.? 172.21.15.78:0/2375551490' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c36ef007-5c1b-4aa8-bee6-db97e9852730"}]': finished 2023-12-21T03:45:14.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:14 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:13.688631+0000 mon.smithi078 (mon.0) 313 : cluster [DBG] osdmap e9: 2 total, 1 up, 2 in 2023-12-21T03:45:14.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:14 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:13.688892+0000 mon.smithi078 (mon.0) 314 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-21T03:45:14.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:14 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:13.689124+0000 mon.smithi078 (mon.0) 315 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:14.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:14 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:14.100895+0000 mon.smithi078 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.78:0/3333428062' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:45:14.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:13.483789+0000 mon.smithi078 (mon.0) 306 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-12-21T03:45:14.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:13.483838+0000 mon.smithi078 (mon.0) 307 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2023-12-21T03:45:14.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:13.483924+0000 mon.smithi078 (mon.0) 308 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-21T03:45:14.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:13.488256+0000 mon.smithi078 (mon.0) 309 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-21T03:45:14.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:13.681741+0000 mon.smithi078 (mon.0) 310 : audit [INF] from='client.? 172.21.15.78:0/2375551490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c36ef007-5c1b-4aa8-bee6-db97e9852730"}]: dispatch 2023-12-21T03:45:14.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:13.688555+0000 mon.smithi078 (mon.0) 311 : cluster [INF] osd.0 [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166] boot 2023-12-21T03:45:14.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:13.688599+0000 mon.smithi078 (mon.0) 312 : audit [INF] from='client.? 172.21.15.78:0/2375551490' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c36ef007-5c1b-4aa8-bee6-db97e9852730"}]': finished 2023-12-21T03:45:14.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:13.688631+0000 mon.smithi078 (mon.0) 313 : cluster [DBG] osdmap e9: 2 total, 1 up, 2 in 2023-12-21T03:45:14.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:13.688892+0000 mon.smithi078 (mon.0) 314 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-21T03:45:14.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:13.689124+0000 mon.smithi078 (mon.0) 315 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:14.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:14 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:14.100895+0000 mon.smithi078 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.78:0/3333428062' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:45:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:13.409574+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:45:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:13.409675+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:45:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:14.230518+0000 mgr.smithi078.zidbgq (mgr.14162) 189 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:15.320648+0000 mon.smithi078 (mon.0) 317 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2023-12-21T03:45:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:15.320740+0000 mon.smithi078 (mon.0) 318 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:13.409574+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:45:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:13.409675+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:45:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:14.230518+0000 mgr.smithi078.zidbgq (mgr.14162) 189 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-21T03:45:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:15.320648+0000 mon.smithi078 (mon.0) 317 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2023-12-21T03:45:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:15.320740+0000 mon.smithi078 (mon.0) 318 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:16.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:16 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:16.230905+0000 mgr.smithi078.zidbgq (mgr.14162) 190 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:17.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:16 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:16.230905+0000 mgr.smithi078.zidbgq (mgr.14162) 190 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:19.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:18.231265+0000 mgr.smithi078.zidbgq (mgr.14162) 191 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:19.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:18.231265+0000 mgr.smithi078.zidbgq (mgr.14162) 191 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:20.474 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:20 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:19.446297+0000 mon.smithi078 (mon.0) 319 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-21T03:45:20.474 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:20 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:19.446932+0000 mon.smithi078 (mon.0) 320 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:20.474 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:20 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:45:19.447379+0000 mgr.smithi078.zidbgq (mgr.14162) 192 : cephadm [INF] Deploying daemon osd.1 on smithi078 2023-12-21T03:45:20.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:20 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:19.446297+0000 mon.smithi078 (mon.0) 319 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-21T03:45:20.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:20 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:19.446932+0000 mon.smithi078 (mon.0) 320 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:20.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:20 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:45:19.447379+0000 mgr.smithi078.zidbgq (mgr.14162) 192 : cephadm [INF] Deploying daemon osd.1 on smithi078 2023-12-21T03:45:21.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:20.231531+0000 mgr.smithi078.zidbgq (mgr.14162) 193 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:21.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:20.231531+0000 mgr.smithi078.zidbgq (mgr.14162) 193 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:21.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.811+0000 7f8459ffb700 1 -- 172.21.15.78:0/3287388580 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f84640fde50 con 0x7f845005c2a0 2023-12-21T03:45:21.818 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 1 on host 'smithi078' 2023-12-21T03:45:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.814+0000 7f846b947700 1 -- 172.21.15.78:0/3287388580 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f845005c2a0 msgr2=0x7f845005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.814+0000 7f846b947700 1 --2- 172.21.15.78:0/3287388580 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f845005c2a0 0x7f845005e760 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f8460004a70 tx=0x7f846000b040).stop 2023-12-21T03:45:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.814+0000 7f846b947700 1 -- 172.21.15.78:0/3287388580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640f5bb0 msgr2=0x7f8464109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.814+0000 7f846b947700 1 --2- 172.21.15.78:0/3287388580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640f5bb0 0x7f8464109c70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f8454007a60 tx=0x7f8454009b30).stop 2023-12-21T03:45:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.814+0000 7f846b947700 1 -- 172.21.15.78:0/3287388580 shutdown_connections 2023-12-21T03:45:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.814+0000 7f846b947700 1 --2- 172.21.15.78:0/3287388580 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f845005c2a0 0x7f845005e760 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.814+0000 7f846b947700 1 --2- 172.21.15.78:0/3287388580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640f5bb0 0x7f8464109c70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.814+0000 7f846b947700 1 --2- 172.21.15.78:0/3287388580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84640f6550 0x7f846410c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.814+0000 7f846b947700 1 -- 172.21.15.78:0/3287388580 >> 172.21.15.78:0/3287388580 conn(0x7f84640f04c0 msgr2=0x7f84640fa450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:45:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.814+0000 7f846b947700 1 -- 172.21.15.78:0/3287388580 shutdown_connections 2023-12-21T03:45:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:21.814+0000 7f846b947700 1 -- 172.21.15.78:0/3287388580 wait complete. 2023-12-21T03:45:22.156 DEBUG:teuthology.orchestra.run.smithi078:osd.1> sudo journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.1.service 2023-12-21T03:45:22.158 INFO:tasks.cephadm:Deploying osd.2 on smithi078 with /dev/vg_nvme/lv_2... 2023-12-21T03:45:22.158 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-21T03:45:22.187 INFO:journalctl@ceph.osd.1.smithi078.stdout:-- Logs begin at Thu 2023-12-21 03:26:39 UTC. -- 2023-12-21T03:45:22.508 INFO:journalctl@ceph.osd.1.smithi078.stdout:Dec 21 03:45:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.1[123909]: debug 2023-12-21T03:45:22.281+0000 7f24213f5080 -1 Falling back to public interface 2023-12-21T03:45:23.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:21.808287+0000 mon.smithi078 (mon.0) 321 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:23.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:21.812039+0000 mon.smithi078 (mon.0) 322 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:23.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:22.231950+0000 mgr.smithi078.zidbgq (mgr.14162) 194 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:23.265 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:21.808287+0000 mon.smithi078 (mon.0) 321 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:23.265 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:21.812039+0000 mon.smithi078 (mon.0) 322 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:23.265 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:22.231950+0000 mgr.smithi078.zidbgq (mgr.14162) 194 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:23.566 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-21T03:45:23.567 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-21T03:45:23.567 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-12-21T03:45:23.567 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-12-21T03:45:23.567 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124611 s, 841 MB/s 2023-12-21T03:45:23.567 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 2023-12-21T03:45:23.567 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-12-21T03:45:23.904 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_2 2023-12-21T03:45:25.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.020+0000 7f559f59e700 1 -- 172.21.15.78:0/3724058870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a00fba60 msgr2=0x7f55a00fde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:25.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.020+0000 7f559f59e700 1 --2- 172.21.15.78:0/3724058870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a00fba60 0x7f55a00fde50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5594004660 tx=0x7f5594009b30).stop 2023-12-21T03:45:25.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.020+0000 7f559f59e700 1 -- 172.21.15.78:0/3724058870 shutdown_connections 2023-12-21T03:45:25.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.020+0000 7f559f59e700 1 --2- 172.21.15.78:0/3724058870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a00fba60 0x7f55a00fde50 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.020+0000 7f559f59e700 1 --2- 172.21.15.78:0/3724058870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55a00602b0 0x7f55a00fb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.020+0000 7f559f59e700 1 -- 172.21.15.78:0/3724058870 >> 172.21.15.78:0/3724058870 conn(0x7f55a00f0b90 msgr2=0x7f55a00f2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.020+0000 7f559f59e700 1 -- 172.21.15.78:0/3724058870 shutdown_connections 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.020+0000 7f559f59e700 1 -- 172.21.15.78:0/3724058870 wait complete. 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559f59e700 1 Processor -- start 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559f59e700 1 -- start start 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559f59e700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55a00602b0 0x7f55a0108f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559f59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a00fba60 0x7f55a0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559f59e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a01099e0 con 0x7f55a00fba60 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559f59e700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a0109b20 con 0x7f55a00602b0 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559dd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a00fba60 0x7f55a0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:45:25.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559dd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a00fba60 0x7f55a0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44978/0 (socket says 172.21.15.78:44978) 2023-12-21T03:45:25.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559dd9b700 1 -- 172.21.15.78:0/2207533351 learned_addr learned my addr 172.21.15.78:0/2207533351 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:45:25.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559dd9b700 1 -- 172.21.15.78:0/2207533351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55a00602b0 msgr2=0x7f55a0108f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:25.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559dd9b700 1 --2- 172.21.15.78:0/2207533351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55a00602b0 0x7f55a0108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:45:25.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559dd9b700 1 -- 172.21.15.78:0/2207533351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5594005040 con 0x7f55a00fba60 2023-12-21T03:45:25.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f559dd9b700 1 --2- 172.21.15.78:0/2207533351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a00fba60 0x7f55a0109430 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f5594004e30 tx=0x7f5594009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:45:25.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.021+0000 7f558effd700 1 -- 172.21.15.78:0/2207533351 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5594018070 con 0x7f55a00fba60 2023-12-21T03:45:25.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.022+0000 7f558effd700 1 -- 172.21.15.78:0/2207533351 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5594013410 con 0x7f55a00fba60 2023-12-21T03:45:25.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.022+0000 7f559f59e700 1 -- 172.21.15.78:0/2207533351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a010a460 con 0x7f55a00fba60 2023-12-21T03:45:25.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.022+0000 7f558effd700 1 -- 172.21.15.78:0/2207533351 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f559401c3c0 con 0x7f55a00fba60 2023-12-21T03:45:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.022+0000 7f559f59e700 1 -- 172.21.15.78:0/2207533351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a010a9c0 con 0x7f55a00fba60 2023-12-21T03:45:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.022+0000 7f559f59e700 1 -- 172.21.15.78:0/2207533351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a0045ad0 con 0x7f55a00fba60 2023-12-21T03:45:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.023+0000 7f558effd700 1 -- 172.21.15.78:0/2207533351 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f559401d420 con 0x7f55a00fba60 2023-12-21T03:45:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.023+0000 7f558effd700 1 --2- 172.21.15.78:0/2207533351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f558805c300 0x7f558805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:45:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.023+0000 7f558effd700 1 -- 172.21.15.78:0/2207533351 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1997+0+0 (secure 0 0 0) 0x7f5594076e20 con 0x7f55a00fba60 2023-12-21T03:45:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.023+0000 7f559e59c700 1 --2- 172.21.15.78:0/2207533351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f558805c300 0x7f558805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:45:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.024+0000 7f559e59c700 1 --2- 172.21.15.78:0/2207533351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f558805c300 0x7f558805e7c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5590004a10 tx=0x7f55900099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:45:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.027+0000 7f558effd700 1 -- 172.21.15.78:0/2207533351 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f559404a820 con 0x7f55a00fba60 2023-12-21T03:45:25.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:25.205+0000 7f559f59e700 1 -- 172.21.15.78:0/2207533351 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f55a010abe0 con 0x7f558805c300 2023-12-21T03:45:25.540 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:24.232388+0000 mgr.smithi078.zidbgq (mgr.14162) 195 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:25.541 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:25.208200+0000 mon.smithi078 (mon.0) 323 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:45:25.541 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:25.209383+0000 mon.smithi078 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:45:25.541 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:25.209826+0000 mon.smithi078 (mon.0) 325 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:24.232388+0000 mgr.smithi078.zidbgq (mgr.14162) 195 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:25.208200+0000 mon.smithi078 (mon.0) 323 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:45:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:25.209383+0000 mon.smithi078 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:45:25.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:25.209826+0000 mon.smithi078 (mon.0) 325 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:26.028 INFO:journalctl@ceph.osd.1.smithi078.stdout:Dec 21 03:45:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.1[123909]: debug 2023-12-21T03:45:25.776+0000 7f24213f5080 -1 osd.1 0 log_to_monitors {default=true} 2023-12-21T03:45:26.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:25.207471+0000 mgr.smithi078.zidbgq (mgr.14162) 196 : audit [DBG] from='client.14428 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:45:26.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:45:25.209209+0000 mgr.smithi078.zidbgq (mgr.14162) 197 : cephadm [INF] Applying service osd.None on host smithi078... 2023-12-21T03:45:26.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:25.780268+0000 mon.smithi078 (mon.0) 326 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-21T03:45:26.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:25.207471+0000 mgr.smithi078.zidbgq (mgr.14162) 196 : audit [DBG] from='client.14428 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:45:26.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:45:25.209209+0000 mgr.smithi078.zidbgq (mgr.14162) 197 : cephadm [INF] Applying service osd.None on host smithi078... 2023-12-21T03:45:26.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:25.780268+0000 mon.smithi078 (mon.0) 326 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-21T03:45:27.316 INFO:journalctl@ceph.osd.1.smithi078.stdout:Dec 21 03:45:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.1[123909]: debug 2023-12-21T03:45:26.827+0000 7f2417966700 -1 osd.1 0 waiting for initial osdmap 2023-12-21T03:45:27.316 INFO:journalctl@ceph.osd.1.smithi078.stdout:Dec 21 03:45:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.1[123909]: debug 2023-12-21T03:45:26.831+0000 7f24100da700 -1 osd.1 12 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2023-12-21T03:45:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:26.232674+0000 mgr.smithi078.zidbgq (mgr.14162) 198 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:26.297653+0000 mon.smithi078 (mon.0) 327 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-21T03:45:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:26.297704+0000 mon.smithi078 (mon.0) 328 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2023-12-21T03:45:27.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:26.297931+0000 mon.smithi078 (mon.0) 329 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:27.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:26.298426+0000 mon.smithi078 (mon.0) 330 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-12-21T03:45:27.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:26.821244+0000 mon.smithi078 (mon.0) 331 : audit [INF] from='client.? 172.21.15.78:0/2947129792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f46dcec-487c-4542-9726-78e82a92d85e"}]: dispatch 2023-12-21T03:45:27.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:26.825151+0000 mon.smithi078 (mon.0) 332 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-12-21T03:45:27.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:26.825205+0000 mon.smithi078 (mon.0) 333 : audit [INF] from='client.? 172.21.15.78:0/2947129792' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f46dcec-487c-4542-9726-78e82a92d85e"}]': finished 2023-12-21T03:45:27.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:26.825253+0000 mon.smithi078 (mon.0) 334 : cluster [DBG] osdmap e12: 3 total, 1 up, 3 in 2023-12-21T03:45:27.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:26.825355+0000 mon.smithi078 (mon.0) 335 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:27.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:26.825690+0000 mon.smithi078 (mon.0) 336 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:27.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:26.828408+0000 mon.smithi078 (mon.0) 337 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:27.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:27.230021+0000 mon.smithi078 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.78:0/3738369528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:45:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:26.232674+0000 mgr.smithi078.zidbgq (mgr.14162) 198 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-12-21T03:45:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:26.297653+0000 mon.smithi078 (mon.0) 327 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-21T03:45:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:26.297704+0000 mon.smithi078 (mon.0) 328 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2023-12-21T03:45:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:26.297931+0000 mon.smithi078 (mon.0) 329 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:26.298426+0000 mon.smithi078 (mon.0) 330 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-12-21T03:45:27.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:26.821244+0000 mon.smithi078 (mon.0) 331 : audit [INF] from='client.? 172.21.15.78:0/2947129792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f46dcec-487c-4542-9726-78e82a92d85e"}]: dispatch 2023-12-21T03:45:27.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:26.825151+0000 mon.smithi078 (mon.0) 332 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-12-21T03:45:27.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:26.825205+0000 mon.smithi078 (mon.0) 333 : audit [INF] from='client.? 172.21.15.78:0/2947129792' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f46dcec-487c-4542-9726-78e82a92d85e"}]': finished 2023-12-21T03:45:27.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:26.825253+0000 mon.smithi078 (mon.0) 334 : cluster [DBG] osdmap e12: 3 total, 1 up, 3 in 2023-12-21T03:45:27.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:26.825355+0000 mon.smithi078 (mon.0) 335 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:27.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:26.825690+0000 mon.smithi078 (mon.0) 336 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:27.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:26.828408+0000 mon.smithi078 (mon.0) 337 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:27.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:27.230021+0000 mon.smithi078 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.78:0/3738369528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:45:29.265 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:26.800437+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:45:29.265 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:26.800569+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:45:29.266 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:27.826659+0000 mon.smithi078 (mon.0) 339 : cluster [INF] osd.1 [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419] boot 2023-12-21T03:45:29.266 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:27.826683+0000 mon.smithi078 (mon.0) 340 : cluster [DBG] osdmap e13: 3 total, 2 up, 3 in 2023-12-21T03:45:29.266 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:27.826958+0000 mon.smithi078 (mon.0) 341 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:29.266 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:27.827228+0000 mon.smithi078 (mon.0) 342 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:29.266 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:28.233152+0000 mgr.smithi078.zidbgq (mgr.14162) 199 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:29.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:26.800437+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:45:29.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:26.800569+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:45:29.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:27.826659+0000 mon.smithi078 (mon.0) 339 : cluster [INF] osd.1 [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419] boot 2023-12-21T03:45:29.317 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:27.826683+0000 mon.smithi078 (mon.0) 340 : cluster [DBG] osdmap e13: 3 total, 2 up, 3 in 2023-12-21T03:45:29.317 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:27.826958+0000 mon.smithi078 (mon.0) 341 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-21T03:45:29.317 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:27.827228+0000 mon.smithi078 (mon.0) 342 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:29.317 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:28.233152+0000 mgr.smithi078.zidbgq (mgr.14162) 199 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:30.265 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:28.832162+0000 mon.smithi078 (mon.0) 343 : cluster [DBG] osdmap e14: 3 total, 2 up, 3 in 2023-12-21T03:45:30.265 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:28.832500+0000 mon.smithi078 (mon.0) 344 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:30.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:28.832162+0000 mon.smithi078 (mon.0) 343 : cluster [DBG] osdmap e14: 3 total, 2 up, 3 in 2023-12-21T03:45:30.316 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:28.832500+0000 mon.smithi078 (mon.0) 344 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:31.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:30.233538+0000 mgr.smithi078.zidbgq (mgr.14162) 200 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:31.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:30.546485+0000 mon.smithi078 (mon.0) 345 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:45:31.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:30.552966+0000 mon.smithi078 (mon.0) 346 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:45:31.265 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:30.233538+0000 mgr.smithi078.zidbgq (mgr.14162) 200 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:31.265 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:30.546485+0000 mon.smithi078 (mon.0) 345 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:45:31.265 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:30.552966+0000 mon.smithi078 (mon.0) 346 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:45:33.428 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:32.233970+0000 mgr.smithi078.zidbgq (mgr.14162) 201 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:33.428 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:32.437957+0000 mon.smithi078 (mon.0) 347 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-21T03:45:33.428 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:32.438506+0000 mon.smithi078 (mon.0) 348 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:33.428 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:45:32.438923+0000 mgr.smithi078.zidbgq (mgr.14162) 202 : cephadm [INF] Deploying daemon osd.2 on smithi078 2023-12-21T03:45:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:32.233970+0000 mgr.smithi078.zidbgq (mgr.14162) 201 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:32.437957+0000 mon.smithi078 (mon.0) 347 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-21T03:45:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:32.438506+0000 mon.smithi078 (mon.0) 348 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:45:32.438923+0000 mgr.smithi078.zidbgq (mgr.14162) 202 : cephadm [INF] Deploying daemon osd.2 on smithi078 2023-12-21T03:45:34.723 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 2 on host 'smithi078' 2023-12-21T03:45:34.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.720+0000 7f558effd700 1 -- 172.21.15.78:0/2207533351 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f55a010abe0 con 0x7f558805c300 2023-12-21T03:45:34.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.722+0000 7f559f59e700 1 -- 172.21.15.78:0/2207533351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f558805c300 msgr2=0x7f558805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:34.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.722+0000 7f559f59e700 1 --2- 172.21.15.78:0/2207533351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f558805c300 0x7f558805e7c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5590004a10 tx=0x7f55900099e0).stop 2023-12-21T03:45:34.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.722+0000 7f559f59e700 1 -- 172.21.15.78:0/2207533351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a00fba60 msgr2=0x7f55a0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:34.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.722+0000 7f559f59e700 1 --2- 172.21.15.78:0/2207533351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a00fba60 0x7f55a0109430 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f5594004e30 tx=0x7f5594009b30).stop 2023-12-21T03:45:34.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.722+0000 7f559f59e700 1 -- 172.21.15.78:0/2207533351 shutdown_connections 2023-12-21T03:45:34.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.722+0000 7f559f59e700 1 --2- 172.21.15.78:0/2207533351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f558805c300 0x7f558805e7c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:34.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.722+0000 7f559f59e700 1 --2- 172.21.15.78:0/2207533351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55a00602b0 0x7f55a0108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:45:34.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.722+0000 7f559f59e700 1 --2- 172.21.15.78:0/2207533351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a00fba60 0x7f55a0109430 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:34.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.722+0000 7f559f59e700 1 -- 172.21.15.78:0/2207533351 >> 172.21.15.78:0/2207533351 conn(0x7f55a00f0b90 msgr2=0x7f55a00f2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:45:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.725+0000 7f559f59e700 1 -- 172.21.15.78:0/2207533351 shutdown_connections 2023-12-21T03:45:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:45:34.725+0000 7f559f59e700 1 -- 172.21.15.78:0/2207533351 wait complete. 2023-12-21T03:45:35.049 DEBUG:teuthology.orchestra.run.smithi078:osd.2> sudo journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.2.service 2023-12-21T03:45:35.051 INFO:tasks.cephadm:Deploying osd.3 on smithi148 with /dev/vg_nvme/lv_4... 2023-12-21T03:45:35.052 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-21T03:45:35.081 INFO:journalctl@ceph.osd.2.smithi078.stdout:-- Logs begin at Thu 2023-12-21 03:26:39 UTC. -- 2023-12-21T03:45:35.290 INFO:journalctl@ceph.osd.2.smithi078.stdout:Dec 21 03:45:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.2[127059]: debug 2023-12-21T03:45:35.174+0000 7fb06a327080 -1 Falling back to public interface 2023-12-21T03:45:35.370 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:34.234293+0000 mgr.smithi078.zidbgq (mgr.14162) 203 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:35.370 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:34.715679+0000 mon.smithi078 (mon.0) 349 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:35.370 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:34.720448+0000 mon.smithi078 (mon.0) 350 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:35.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:34.234293+0000 mgr.smithi078.zidbgq (mgr.14162) 203 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:35.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:34.715679+0000 mon.smithi078 (mon.0) 349 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:35.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:34.720448+0000 mon.smithi078 (mon.0) 350 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:45:36.486 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-21T03:45:36.486 INFO:teuthology.orchestra.run.smithi148.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-21T03:45:36.486 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10+0 records in 2023-12-21T03:45:36.486 INFO:teuthology.orchestra.run.smithi148.stderr:10+0 records out 2023-12-21T03:45:36.486 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119553 s, 877 MB/s 2023-12-21T03:45:36.486 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping successful for: 2023-12-21T03:45:36.839 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi148:vg_nvme/lv_4 2023-12-21T03:45:37.421 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:36.234777+0000 mgr.smithi078.zidbgq (mgr.14162) 204 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:37.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:36.234777+0000 mgr.smithi078.zidbgq (mgr.14162) 204 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:37.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.943+0000 7f3e45967700 1 -- 172.21.15.148:0/3506268159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e400f6860 msgr2=0x7f3e400f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:37.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.943+0000 7f3e45967700 1 --2- 172.21.15.148:0/3506268159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e400f6860 0x7f3e400f6cc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f3e34004660 tx=0x7f3e34009b30).stop 2023-12-21T03:45:37.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.943+0000 7f3e45967700 1 -- 172.21.15.148:0/3506268159 shutdown_connections 2023-12-21T03:45:37.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.943+0000 7f3e45967700 1 --2- 172.21.15.148:0/3506268159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e400f6860 0x7f3e400f6cc0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:37.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.943+0000 7f3e45967700 1 --2- 172.21.15.148:0/3506268159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e400f55c0 0x7f3e400f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:37.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.943+0000 7f3e45967700 1 -- 172.21.15.148:0/3506268159 >> 172.21.15.148:0/3506268159 conn(0x7f3e400f0d40 msgr2=0x7f3e400f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:45:37.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.943+0000 7f3e45967700 1 -- 172.21.15.148:0/3506268159 shutdown_connections 2023-12-21T03:45:37.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.943+0000 7f3e45967700 1 -- 172.21.15.148:0/3506268159 wait complete. 2023-12-21T03:45:37.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.944+0000 7f3e45967700 1 Processor -- start 2023-12-21T03:45:37.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.944+0000 7f3e45967700 1 -- start start 2023-12-21T03:45:37.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.944+0000 7f3e45967700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e400f55c0 0x7f3e40109cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:45:37.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.944+0000 7f3e45967700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e400f6860 0x7f3e4010c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:45:37.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.944+0000 7f3e45967700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400604e0 con 0x7f3e400f6860 2023-12-21T03:45:37.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.944+0000 7f3e45967700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e40060620 con 0x7f3e400f55c0 2023-12-21T03:45:37.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.944+0000 7f3e3effd700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e400f55c0 0x7f3e40109cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.944+0000 7f3e3effd700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e400f55c0 0x7f3e40109cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37704/0 (socket says 172.21.15.148:37704) 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.944+0000 7f3e3effd700 1 -- 172.21.15.148:0/2015495900 learned_addr learned my addr 172.21.15.148:0/2015495900 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.945+0000 7f3e3e7fc700 1 --2- 172.21.15.148:0/2015495900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e400f6860 0x7f3e4010c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.945+0000 7f3e3effd700 1 -- 172.21.15.148:0/2015495900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e400f6860 msgr2=0x7f3e4010c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.945+0000 7f3e3effd700 1 --2- 172.21.15.148:0/2015495900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e400f6860 0x7f3e4010c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.945+0000 7f3e3effd700 1 -- 172.21.15.148:0/2015495900 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e34005040 con 0x7f3e400f55c0 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.945+0000 7f3e3effd700 1 --2- 172.21.15.148:0/2015495900 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e400f55c0 0x7f3e40109cb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f3e30002700 tx=0x7f3e30009d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.945+0000 7f3e277fe700 1 -- 172.21.15.148:0/2015495900 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e300079c0 con 0x7f3e400f55c0 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.945+0000 7f3e277fe700 1 -- 172.21.15.148:0/2015495900 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e3000f040 con 0x7f3e400f55c0 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.945+0000 7f3e45967700 1 -- 172.21.15.148:0/2015495900 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4010c760 con 0x7f3e400f55c0 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.945+0000 7f3e277fe700 1 -- 172.21.15.148:0/2015495900 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e30014860 con 0x7f3e400f55c0 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.945+0000 7f3e45967700 1 -- 172.21.15.148:0/2015495900 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4010ccf0 con 0x7f3e400f55c0 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.946+0000 7f3e45967700 1 -- 172.21.15.148:0/2015495900 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e400f49b0 con 0x7f3e400f55c0 2023-12-21T03:45:37.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.946+0000 7f3e277fe700 1 -- 172.21.15.148:0/2015495900 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3e3001b060 con 0x7f3e400f55c0 2023-12-21T03:45:37.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.947+0000 7f3e277fe700 1 --2- 172.21.15.148:0/2015495900 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3e2805c2b0 0x7f3e2805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:45:37.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.947+0000 7f3e277fe700 1 -- 172.21.15.148:0/2015495900 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2429+0+0 (secure 0 0 0) 0x7f3e30075130 con 0x7f3e400f55c0 2023-12-21T03:45:37.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.947+0000 7f3e3e7fc700 1 --2- 172.21.15.148:0/2015495900 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3e2805c2b0 0x7f3e2805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:45:37.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.947+0000 7f3e3e7fc700 1 --2- 172.21.15.148:0/2015495900 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3e2805c2b0 0x7f3e2805e770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f3e34009f50 tx=0x7f3e34009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:45:37.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:37.951+0000 7f3e277fe700 1 -- 172.21.15.148:0/2015495900 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e30048a00 con 0x7f3e400f55c0 2023-12-21T03:45:38.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:45:38.124+0000 7f3e45967700 1 -- 172.21.15.148:0/2015495900 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e40057f10 con 0x7f3e2805c2b0 2023-12-21T03:45:38.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:38.127361+0000 mon.smithi078 (mon.0) 351 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:45:38.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:38.129065+0000 mon.smithi078 (mon.0) 352 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:45:38.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:38.129658+0000 mon.smithi078 (mon.0) 353 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:38.591 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:38 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:38.127361+0000 mon.smithi078 (mon.0) 351 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:45:38.591 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:38 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:38.129065+0000 mon.smithi078 (mon.0) 352 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:45:38.592 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:38 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:38.129658+0000 mon.smithi078 (mon.0) 353 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:45:39.066 INFO:journalctl@ceph.osd.2.smithi078.stdout:Dec 21 03:45:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.2[127059]: debug 2023-12-21T03:45:38.625+0000 7fb06a327080 -1 osd.2 0 log_to_monitors {default=true} 2023-12-21T03:45:39.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:38.126486+0000 mgr.smithi078.zidbgq (mgr.14162) 205 : audit [DBG] from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:45:39.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:45:38.128806+0000 mgr.smithi078.zidbgq (mgr.14162) 206 : cephadm [INF] Applying service osd.None on host smithi148... 2023-12-21T03:45:39.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:38.235197+0000 mgr.smithi078.zidbgq (mgr.14162) 207 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:39.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:38.630271+0000 mon.smithi148 (mon.1) 2 : audit [INF] from='osd.2 [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-21T03:45:39.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:38.630676+0000 mon.smithi078 (mon.0) 354 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-21T03:45:39.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:38.126486+0000 mgr.smithi078.zidbgq (mgr.14162) 205 : audit [DBG] from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:45:39.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:45:38.128806+0000 mgr.smithi078.zidbgq (mgr.14162) 206 : cephadm [INF] Applying service osd.None on host smithi148... 2023-12-21T03:45:39.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:38.235197+0000 mgr.smithi078.zidbgq (mgr.14162) 207 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:39.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:38.630271+0000 mon.smithi148 (mon.1) 2 : audit [INF] from='osd.2 [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-21T03:45:39.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:38.630676+0000 mon.smithi078 (mon.0) 354 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-21T03:45:40.566 INFO:journalctl@ceph.osd.2.smithi078.stdout:Dec 21 03:45:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.2[127059]: debug 2023-12-21T03:45:40.310+0000 7fb060898700 -1 osd.2 0 waiting for initial osdmap 2023-12-21T03:45:40.566 INFO:journalctl@ceph.osd.2.smithi078.stdout:Dec 21 03:45:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.2[127059]: debug 2023-12-21T03:45:40.318+0000 7fb05b811700 -1 osd.2 16 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2023-12-21T03:45:40.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:39.306623+0000 mon.smithi078 (mon.0) 355 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-21T03:45:40.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:39.306681+0000 mon.smithi078 (mon.0) 356 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2023-12-21T03:45:40.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:39.307417+0000 mon.smithi148 (mon.1) 3 : audit [INF] from='osd.2 [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-12-21T03:45:40.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:39.307682+0000 mon.smithi078 (mon.0) 357 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:40.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:39.308027+0000 mon.smithi078 (mon.0) 358 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-12-21T03:45:40.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:40.308124+0000 mon.smithi078 (mon.0) 359 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-12-21T03:45:40.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:40.308189+0000 mon.smithi078 (mon.0) 360 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2023-12-21T03:45:40.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:40.308299+0000 mon.smithi078 (mon.0) 361 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:40.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:40 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:39.306623+0000 mon.smithi078 (mon.0) 355 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-21T03:45:40.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:40 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:39.306681+0000 mon.smithi078 (mon.0) 356 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2023-12-21T03:45:40.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:40 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:39.307417+0000 mon.smithi148 (mon.1) 3 : audit [INF] from='osd.2 [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-12-21T03:45:40.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:40 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:39.307682+0000 mon.smithi078 (mon.0) 357 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:40.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:40 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:39.308027+0000 mon.smithi078 (mon.0) 358 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-12-21T03:45:40.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:40 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:40.308124+0000 mon.smithi078 (mon.0) 359 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-12-21T03:45:40.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:40 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:40.308189+0000 mon.smithi078 (mon.0) 360 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2023-12-21T03:45:40.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:40 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:40.308299+0000 mon.smithi078 (mon.0) 361 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:41.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:40.235507+0000 mgr.smithi078.zidbgq (mgr.14162) 208 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:41.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:40.311898+0000 mon.smithi078 (mon.0) 362 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:41.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:41.311842+0000 mon.smithi078 (mon.0) 363 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:40.235507+0000 mgr.smithi078.zidbgq (mgr.14162) 208 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:40.311898+0000 mon.smithi078 (mon.0) 362 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:41.311842+0000 mon.smithi078 (mon.0) 363 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:42.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:42 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:39.647944+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:45:42.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:42 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:39.648048+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:45:42.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:42 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:41.323996+0000 mon.smithi078 (mon.0) 364 : cluster [INF] osd.2 [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027] boot 2023-12-21T03:45:42.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:42 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:41.324032+0000 mon.smithi078 (mon.0) 365 : cluster [DBG] osdmap e17: 3 total, 3 up, 3 in 2023-12-21T03:45:42.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:42 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:45:41.324405+0000 mon.smithi078 (mon.0) 366 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:42.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:39.647944+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:45:42.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:39.648048+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:45:42.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:41.323996+0000 mon.smithi078 (mon.0) 364 : cluster [INF] osd.2 [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027] boot 2023-12-21T03:45:42.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:41.324032+0000 mon.smithi078 (mon.0) 365 : cluster [DBG] osdmap e17: 3 total, 3 up, 3 in 2023-12-21T03:45:42.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:45:41.324405+0000 mon.smithi078 (mon.0) 366 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-21T03:45:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:42.235810+0000 mgr.smithi078.zidbgq (mgr.14162) 209 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:42.336701+0000 mon.smithi078 (mon.0) 367 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2023-12-21T03:45:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:42.235810+0000 mgr.smithi078.zidbgq (mgr.14162) 209 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-12-21T03:45:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:42.336701+0000 mon.smithi078 (mon.0) 367 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2023-12-21T03:45:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:44.236201+0000 mgr.smithi078.zidbgq (mgr.14162) 210 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:44.236201+0000 mgr.smithi078.zidbgq (mgr.14162) 210 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:46.236584+0000 mgr.smithi078.zidbgq (mgr.14162) 211 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:46.236584+0000 mgr.smithi078.zidbgq (mgr.14162) 211 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:48.236985+0000 mgr.smithi078.zidbgq (mgr.14162) 212 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:49.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:48.236985+0000 mgr.smithi078.zidbgq (mgr.14162) 212 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:50.237455+0000 mgr.smithi078.zidbgq (mgr.14162) 213 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:50.237455+0000 mgr.smithi078.zidbgq (mgr.14162) 213 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:53.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:52.237785+0000 mgr.smithi078.zidbgq (mgr.14162) 214 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:52.237785+0000 mgr.smithi078.zidbgq (mgr.14162) 214 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:54.238260+0000 mgr.smithi078.zidbgq (mgr.14162) 215 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:54.238260+0000 mgr.smithi078.zidbgq (mgr.14162) 215 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:56.238601+0000 mgr.smithi078.zidbgq (mgr.14162) 216 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:56.238601+0000 mgr.smithi078.zidbgq (mgr.14162) 216 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:45:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:45:58.238991+0000 mgr.smithi078.zidbgq (mgr.14162) 217 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:45:59.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:45:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:45:58.238991+0000 mgr.smithi078.zidbgq (mgr.14162) 217 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:00.239326+0000 mgr.smithi078.zidbgq (mgr.14162) 218 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:00.239326+0000 mgr.smithi078.zidbgq (mgr.14162) 218 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:03.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:02.239590+0000 mgr.smithi078.zidbgq (mgr.14162) 219 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:02.239590+0000 mgr.smithi078.zidbgq (mgr.14162) 219 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:04.240025+0000 mgr.smithi078.zidbgq (mgr.14162) 220 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:05.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:04.240025+0000 mgr.smithi078.zidbgq (mgr.14162) 220 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:07.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:06.240489+0000 mgr.smithi078.zidbgq (mgr.14162) 221 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:06.240489+0000 mgr.smithi078.zidbgq (mgr.14162) 221 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:09.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:08.240904+0000 mgr.smithi078.zidbgq (mgr.14162) 222 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:09.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:08.240904+0000 mgr.smithi078.zidbgq (mgr.14162) 222 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:10.241348+0000 mgr.smithi078.zidbgq (mgr.14162) 223 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:10.241348+0000 mgr.smithi078.zidbgq (mgr.14162) 223 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:13.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:12.241811+0000 mgr.smithi078.zidbgq (mgr.14162) 224 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:12.241811+0000 mgr.smithi078.zidbgq (mgr.14162) 224 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:14.242310+0000 mgr.smithi078.zidbgq (mgr.14162) 225 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:14.242310+0000 mgr.smithi078.zidbgq (mgr.14162) 225 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:17.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:16.242658+0000 mgr.smithi078.zidbgq (mgr.14162) 226 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:17.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:16.242658+0000 mgr.smithi078.zidbgq (mgr.14162) 226 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:19.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:18.243057+0000 mgr.smithi078.zidbgq (mgr.14162) 227 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:19.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:18.243057+0000 mgr.smithi078.zidbgq (mgr.14162) 227 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:21.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:20.243422+0000 mgr.smithi078.zidbgq (mgr.14162) 228 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:21.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:20.243422+0000 mgr.smithi078.zidbgq (mgr.14162) 228 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:23.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:22.243860+0000 mgr.smithi078.zidbgq (mgr.14162) 229 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:22.243860+0000 mgr.smithi078.zidbgq (mgr.14162) 229 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:24.244268+0000 mgr.smithi078.zidbgq (mgr.14162) 230 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:24.244268+0000 mgr.smithi078.zidbgq (mgr.14162) 230 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:26.244532+0000 mgr.smithi078.zidbgq (mgr.14162) 231 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:26.244532+0000 mgr.smithi078.zidbgq (mgr.14162) 231 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:29.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:28.244905+0000 mgr.smithi078.zidbgq (mgr.14162) 232 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:28.244905+0000 mgr.smithi078.zidbgq (mgr.14162) 232 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:30.245223+0000 mgr.smithi078.zidbgq (mgr.14162) 233 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:46:30.553980+0000 mon.smithi078 (mon.0) 368 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:46:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:46:30.554267+0000 mon.smithi078 (mon.0) 369 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:46:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:30.245223+0000 mgr.smithi078.zidbgq (mgr.14162) 233 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:46:30.553980+0000 mon.smithi078 (mon.0) 368 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:46:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:46:30.554267+0000 mon.smithi078 (mon.0) 369 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:46:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:32.245657+0000 mgr.smithi078.zidbgq (mgr.14162) 234 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:33.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:32.245657+0000 mgr.smithi078.zidbgq (mgr.14162) 234 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:34.246149+0000 mgr.smithi078.zidbgq (mgr.14162) 235 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:35.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:34.246149+0000 mgr.smithi078.zidbgq (mgr.14162) 235 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:37.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:36.246467+0000 mgr.smithi078.zidbgq (mgr.14162) 236 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:37.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:36.246467+0000 mgr.smithi078.zidbgq (mgr.14162) 236 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:39.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:38.246818+0000 mgr.smithi078.zidbgq (mgr.14162) 237 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:39.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:38.246818+0000 mgr.smithi078.zidbgq (mgr.14162) 237 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:41.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:40.247260+0000 mgr.smithi078.zidbgq (mgr.14162) 238 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:40.247260+0000 mgr.smithi078.zidbgq (mgr.14162) 238 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:42.247693+0000 mgr.smithi078.zidbgq (mgr.14162) 239 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:42.247693+0000 mgr.smithi078.zidbgq (mgr.14162) 239 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:44.248081+0000 mgr.smithi078.zidbgq (mgr.14162) 240 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:44.248081+0000 mgr.smithi078.zidbgq (mgr.14162) 240 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:46.248431+0000 mgr.smithi078.zidbgq (mgr.14162) 241 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:46.248431+0000 mgr.smithi078.zidbgq (mgr.14162) 241 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:48.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:46:47.055353+0000 mgr.smithi078.zidbgq (mgr.14162) 242 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi148 ... 2023-12-21T03:46:48.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Verifying port 9100 ... 2023-12-21T03:46:48.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Non-zero exit code 1 from systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:46:48.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service failed because a timeout was exceeded. 2023-12-21T03:46:48.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service" and "journalctl -xe" for details. 2023-12-21T03:46:48.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:46:48.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: main() 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: r = ctx.func(ctx) 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: return func(ctx) 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: ports=daemon_ports) 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:46:48.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: RuntimeError: Failed command: systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: yield (conn, connr) 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: code, '\n'.join(err))) 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi148 ... 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Verifying port 9100 ... 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Non-zero exit code 1 from systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service failed because a timeout was exceeded. 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service" and "journalctl -xe" for details. 2023-12-21T03:46:48.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: main() 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: r = ctx.func(ctx) 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: return func(ctx) 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: ports=daemon_ports) 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-21T03:46:48.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-21T03:46:48.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:46:48.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:46:48.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: RuntimeError: Failed command: systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:46:48.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:46:47.058661+0000 mgr.smithi078.zidbgq (mgr.14162) 243 : cephadm [INF] Deploying daemon prometheus.smithi148 on smithi148 2023-12-21T03:46:48.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:46:47.871820+0000 mon.smithi148 (mon.1) 4 : audit [INF] from='client.? 172.21.15.148:0/1795346996' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a26620c-6292-46a1-b9d8-5228ba82bdb7"}]: dispatch 2023-12-21T03:46:48.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:46:47.872704+0000 mon.smithi078 (mon.0) 370 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a26620c-6292-46a1-b9d8-5228ba82bdb7"}]: dispatch 2023-12-21T03:46:48.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:46:47.879606+0000 mon.smithi078 (mon.0) 371 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a26620c-6292-46a1-b9d8-5228ba82bdb7"}]': finished 2023-12-21T03:46:48.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:47.879661+0000 mon.smithi078 (mon.0) 372 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2023-12-21T03:46:48.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:46:47.879763+0000 mon.smithi078 (mon.0) 373 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:46:48.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:48 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:46:48.285107+0000 mon.smithi148 (mon.1) 5 : audit [DBG] from='client.? 172.21.15.148:0/2810894954' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:46:48.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:46:47.055353+0000 mgr.smithi078.zidbgq (mgr.14162) 242 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi148 ... 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Verifying port 9100 ... 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 1 from systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service failed because a timeout was exceeded. 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service" and "journalctl -xe" for details. 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:46:48.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ports=daemon_ports) 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: Failed command: systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: yield (conn, connr) 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: code, '\n'.join(err))) 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi148 ... 2023-12-21T03:46:48.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Verifying port 9100 ... 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 1 from systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service failed because a timeout was exceeded. 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service" and "journalctl -xe" for details. 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ports=daemon_ports) 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-21T03:46:48.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-21T03:46:48.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:46:48.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:46:48.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: Failed command: systemctl start ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:46:48.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:46:47.058661+0000 mgr.smithi078.zidbgq (mgr.14162) 243 : cephadm [INF] Deploying daemon prometheus.smithi148 on smithi148 2023-12-21T03:46:48.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:46:47.871820+0000 mon.smithi148 (mon.1) 4 : audit [INF] from='client.? 172.21.15.148:0/1795346996' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a26620c-6292-46a1-b9d8-5228ba82bdb7"}]: dispatch 2023-12-21T03:46:48.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:46:47.872704+0000 mon.smithi078 (mon.0) 370 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a26620c-6292-46a1-b9d8-5228ba82bdb7"}]: dispatch 2023-12-21T03:46:48.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:46:47.879606+0000 mon.smithi078 (mon.0) 371 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a26620c-6292-46a1-b9d8-5228ba82bdb7"}]': finished 2023-12-21T03:46:48.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:47.879661+0000 mon.smithi078 (mon.0) 372 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2023-12-21T03:46:48.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:46:47.879763+0000 mon.smithi078 (mon.0) 373 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:46:48.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:48 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:46:48.285107+0000 mon.smithi148 (mon.1) 5 : audit [DBG] from='client.? 172.21.15.148:0/2810894954' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:46:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:48.248730+0000 mgr.smithi078.zidbgq (mgr.14162) 244 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:49.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:48.248730+0000 mgr.smithi078.zidbgq (mgr.14162) 244 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:51.727 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:50.249193+0000 mgr.smithi078.zidbgq (mgr.14162) 245 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:50.249193+0000 mgr.smithi078.zidbgq (mgr.14162) 245 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:53.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:52.249598+0000 mgr.smithi078.zidbgq (mgr.14162) 246 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:52.249598+0000 mgr.smithi078.zidbgq (mgr.14162) 246 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:54.250030+0000 mgr.smithi078.zidbgq (mgr.14162) 247 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:54.250030+0000 mgr.smithi078.zidbgq (mgr.14162) 247 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:56.250367+0000 mgr.smithi078.zidbgq (mgr.14162) 248 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:56.250367+0000 mgr.smithi078.zidbgq (mgr.14162) 248 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:46:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:46:58.250700+0000 mgr.smithi078.zidbgq (mgr.14162) 249 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:46:59.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:46:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:46:58.250700+0000 mgr.smithi078.zidbgq (mgr.14162) 249 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:00.251135+0000 mgr.smithi078.zidbgq (mgr.14162) 250 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:00.251135+0000 mgr.smithi078.zidbgq (mgr.14162) 250 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:03.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:02.251650+0000 mgr.smithi078.zidbgq (mgr.14162) 251 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:02.251650+0000 mgr.smithi078.zidbgq (mgr.14162) 251 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:04.252008+0000 mgr.smithi078.zidbgq (mgr.14162) 252 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:05.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:04.252008+0000 mgr.smithi078.zidbgq (mgr.14162) 252 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:07.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:06.252458+0000 mgr.smithi078.zidbgq (mgr.14162) 253 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:06.252458+0000 mgr.smithi078.zidbgq (mgr.14162) 253 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:09.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:08.252925+0000 mgr.smithi078.zidbgq (mgr.14162) 254 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:09.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:08.252925+0000 mgr.smithi078.zidbgq (mgr.14162) 254 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:10.253352+0000 mgr.smithi078.zidbgq (mgr.14162) 255 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:10.253352+0000 mgr.smithi078.zidbgq (mgr.14162) 255 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:13.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:12.253678+0000 mgr.smithi078.zidbgq (mgr.14162) 256 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:12.253678+0000 mgr.smithi078.zidbgq (mgr.14162) 256 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:14.254002+0000 mgr.smithi078.zidbgq (mgr.14162) 257 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:14.254002+0000 mgr.smithi078.zidbgq (mgr.14162) 257 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:17.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:16.254472+0000 mgr.smithi078.zidbgq (mgr.14162) 258 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:18.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:16.254472+0000 mgr.smithi078.zidbgq (mgr.14162) 258 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:19.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:18.254890+0000 mgr.smithi078.zidbgq (mgr.14162) 259 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:20.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:18.254890+0000 mgr.smithi078.zidbgq (mgr.14162) 259 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:20.255225+0000 mgr.smithi078.zidbgq (mgr.14162) 260 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:22.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:20.255225+0000 mgr.smithi078.zidbgq (mgr.14162) 260 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:22.255498+0000 mgr.smithi078.zidbgq (mgr.14162) 261 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:24.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:22.255498+0000 mgr.smithi078.zidbgq (mgr.14162) 261 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:25.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:24.255832+0000 mgr.smithi078.zidbgq (mgr.14162) 262 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:24.255832+0000 mgr.smithi078.zidbgq (mgr.14162) 262 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:27.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:26.256335+0000 mgr.smithi078.zidbgq (mgr.14162) 263 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:26.256335+0000 mgr.smithi078.zidbgq (mgr.14162) 263 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:29.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:28.256644+0000 mgr.smithi078.zidbgq (mgr.14162) 264 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:29.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:28.256644+0000 mgr.smithi078.zidbgq (mgr.14162) 264 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:31.570 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:30.257020+0000 mgr.smithi078.zidbgq (mgr.14162) 265 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:31.571 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:47:30.555589+0000 mon.smithi078 (mon.0) 374 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:47:31.571 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:47:30.555788+0000 mon.smithi078 (mon.0) 375 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:47:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:30.257020+0000 mgr.smithi078.zidbgq (mgr.14162) 265 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:47:30.555589+0000 mon.smithi078 (mon.0) 374 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:47:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:47:30.555788+0000 mon.smithi078 (mon.0) 375 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:47:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:32.257517+0000 mgr.smithi078.zidbgq (mgr.14162) 266 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:33.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:32.257517+0000 mgr.smithi078.zidbgq (mgr.14162) 266 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:34.257858+0000 mgr.smithi078.zidbgq (mgr.14162) 267 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:35.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:34.257858+0000 mgr.smithi078.zidbgq (mgr.14162) 267 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:37.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:36.258297+0000 mgr.smithi078.zidbgq (mgr.14162) 268 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:36.258297+0000 mgr.smithi078.zidbgq (mgr.14162) 268 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:39.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:38.258747+0000 mgr.smithi078.zidbgq (mgr.14162) 269 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:39.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:38.258747+0000 mgr.smithi078.zidbgq (mgr.14162) 269 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:41.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:40.259228+0000 mgr.smithi078.zidbgq (mgr.14162) 270 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:40.259228+0000 mgr.smithi078.zidbgq (mgr.14162) 270 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:42.259552+0000 mgr.smithi078.zidbgq (mgr.14162) 271 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:42.259552+0000 mgr.smithi078.zidbgq (mgr.14162) 271 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:44.260002+0000 mgr.smithi078.zidbgq (mgr.14162) 272 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:44.260002+0000 mgr.smithi078.zidbgq (mgr.14162) 272 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:46.260339+0000 mgr.smithi078.zidbgq (mgr.14162) 273 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:46.260339+0000 mgr.smithi078.zidbgq (mgr.14162) 273 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:48.260727+0000 mgr.smithi078.zidbgq (mgr.14162) 274 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:49.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:48.260727+0000 mgr.smithi078.zidbgq (mgr.14162) 274 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:50.261132+0000 mgr.smithi078.zidbgq (mgr.14162) 275 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:50.261132+0000 mgr.smithi078.zidbgq (mgr.14162) 275 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:53.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:52.261429+0000 mgr.smithi078.zidbgq (mgr.14162) 276 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:53.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:52.261429+0000 mgr.smithi078.zidbgq (mgr.14162) 276 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:54.261844+0000 mgr.smithi078.zidbgq (mgr.14162) 277 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:54.261844+0000 mgr.smithi078.zidbgq (mgr.14162) 277 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:56.262320+0000 mgr.smithi078.zidbgq (mgr.14162) 278 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:56.262320+0000 mgr.smithi078.zidbgq (mgr.14162) 278 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:47:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:47:58.262694+0000 mgr.smithi078.zidbgq (mgr.14162) 279 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:47:59.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:47:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:47:58.262694+0000 mgr.smithi078.zidbgq (mgr.14162) 279 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:01.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:00.263062+0000 mgr.smithi078.zidbgq (mgr.14162) 280 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:01.828 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:00.263062+0000 mgr.smithi078.zidbgq (mgr.14162) 280 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:03.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:02.263450+0000 mgr.smithi078.zidbgq (mgr.14162) 281 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:02.263450+0000 mgr.smithi078.zidbgq (mgr.14162) 281 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:04.263769+0000 mgr.smithi078.zidbgq (mgr.14162) 282 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:05.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:04.263769+0000 mgr.smithi078.zidbgq (mgr.14162) 282 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:07.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:06.264055+0000 mgr.smithi078.zidbgq (mgr.14162) 283 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:06.264055+0000 mgr.smithi078.zidbgq (mgr.14162) 283 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:09.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:08.264515+0000 mgr.smithi078.zidbgq (mgr.14162) 284 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:09.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:08.264515+0000 mgr.smithi078.zidbgq (mgr.14162) 284 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:10.264961+0000 mgr.smithi078.zidbgq (mgr.14162) 285 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:10.264961+0000 mgr.smithi078.zidbgq (mgr.14162) 285 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:13.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:12.265429+0000 mgr.smithi078.zidbgq (mgr.14162) 286 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:12.265429+0000 mgr.smithi078.zidbgq (mgr.14162) 286 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:14.265749+0000 mgr.smithi078.zidbgq (mgr.14162) 287 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:14.265749+0000 mgr.smithi078.zidbgq (mgr.14162) 287 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:17.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:16.266241+0000 mgr.smithi078.zidbgq (mgr.14162) 288 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:17.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:16.266241+0000 mgr.smithi078.zidbgq (mgr.14162) 288 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:19.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:18.266636+0000 mgr.smithi078.zidbgq (mgr.14162) 289 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:19.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:18.266636+0000 mgr.smithi078.zidbgq (mgr.14162) 289 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:21.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:20.267222+0000 mgr.smithi078.zidbgq (mgr.14162) 290 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:20.267222+0000 mgr.smithi078.zidbgq (mgr.14162) 290 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:23.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:22.267615+0000 mgr.smithi078.zidbgq (mgr.14162) 291 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:22.267615+0000 mgr.smithi078.zidbgq (mgr.14162) 291 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:24.268123+0000 mgr.smithi078.zidbgq (mgr.14162) 292 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:24.268123+0000 mgr.smithi078.zidbgq (mgr.14162) 292 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:26.268458+0000 mgr.smithi078.zidbgq (mgr.14162) 293 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:26.268458+0000 mgr.smithi078.zidbgq (mgr.14162) 293 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:29.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:28.268760+0000 mgr.smithi078.zidbgq (mgr.14162) 294 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:28.268760+0000 mgr.smithi078.zidbgq (mgr.14162) 294 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:30.269123+0000 mgr.smithi078.zidbgq (mgr.14162) 295 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:48:30.556562+0000 mon.smithi078 (mon.0) 376 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:48:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:48:30.556898+0000 mon.smithi078 (mon.0) 377 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:48:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:30.269123+0000 mgr.smithi078.zidbgq (mgr.14162) 295 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:48:30.556562+0000 mon.smithi078 (mon.0) 376 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:48:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:48:30.556898+0000 mon.smithi078 (mon.0) 377 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:48:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:32.269588+0000 mgr.smithi078.zidbgq (mgr.14162) 296 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:33.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:32.269588+0000 mgr.smithi078.zidbgq (mgr.14162) 296 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:34.270040+0000 mgr.smithi078.zidbgq (mgr.14162) 297 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:35.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:34.270040+0000 mgr.smithi078.zidbgq (mgr.14162) 297 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:37.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:36.270472+0000 mgr.smithi078.zidbgq (mgr.14162) 298 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:36.270472+0000 mgr.smithi078.zidbgq (mgr.14162) 298 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:37.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:48:37.966+0000 7f3e267fc700 1 -- 172.21.15.148:0/2015495900 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e180031e0 con 0x7f3e400f55c0 2023-12-21T03:48:37.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:48:37.966+0000 7f3e277fe700 1 -- 172.21.15.148:0/2015495900 <== mon.1 v2:172.21.15.148:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f3e180031e0 con 0x7f3e400f55c0 2023-12-21T03:48:39.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:38.270820+0000 mgr.smithi078.zidbgq (mgr.14162) 299 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:39.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:38.270820+0000 mgr.smithi078.zidbgq (mgr.14162) 299 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:41.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:40.271194+0000 mgr.smithi078.zidbgq (mgr.14162) 300 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:40.271194+0000 mgr.smithi078.zidbgq (mgr.14162) 300 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:42.271600+0000 mgr.smithi078.zidbgq (mgr.14162) 301 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:42.271600+0000 mgr.smithi078.zidbgq (mgr.14162) 301 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:44.272085+0000 mgr.smithi078.zidbgq (mgr.14162) 302 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:44.272085+0000 mgr.smithi078.zidbgq (mgr.14162) 302 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:46.272560+0000 mgr.smithi078.zidbgq (mgr.14162) 303 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:46.272560+0000 mgr.smithi078.zidbgq (mgr.14162) 303 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:48.272978+0000 mgr.smithi078.zidbgq (mgr.14162) 304 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:49.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:48.272978+0000 mgr.smithi078.zidbgq (mgr.14162) 304 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:50.273445+0000 mgr.smithi078.zidbgq (mgr.14162) 305 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:50.273445+0000 mgr.smithi078.zidbgq (mgr.14162) 305 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:53.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:52.273891+0000 mgr.smithi078.zidbgq (mgr.14162) 306 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:52.273891+0000 mgr.smithi078.zidbgq (mgr.14162) 306 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:54.274362+0000 mgr.smithi078.zidbgq (mgr.14162) 307 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:54.274362+0000 mgr.smithi078.zidbgq (mgr.14162) 307 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:56.274823+0000 mgr.smithi078.zidbgq (mgr.14162) 308 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:56.274823+0000 mgr.smithi078.zidbgq (mgr.14162) 308 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:48:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:48:58.275311+0000 mgr.smithi078.zidbgq (mgr.14162) 309 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:48:59.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:48:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:48:58.275311+0000 mgr.smithi078.zidbgq (mgr.14162) 309 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:00.275690+0000 mgr.smithi078.zidbgq (mgr.14162) 310 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:00.275690+0000 mgr.smithi078.zidbgq (mgr.14162) 310 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:03.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:02.276065+0000 mgr.smithi078.zidbgq (mgr.14162) 311 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:02.276065+0000 mgr.smithi078.zidbgq (mgr.14162) 311 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:04.276329+0000 mgr.smithi078.zidbgq (mgr.14162) 312 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:05.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:04.276329+0000 mgr.smithi078.zidbgq (mgr.14162) 312 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:07.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:06.276663+0000 mgr.smithi078.zidbgq (mgr.14162) 313 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:06.276663+0000 mgr.smithi078.zidbgq (mgr.14162) 313 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:09.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:08.277060+0000 mgr.smithi078.zidbgq (mgr.14162) 314 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:09.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:08.277060+0000 mgr.smithi078.zidbgq (mgr.14162) 314 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:10.277462+0000 mgr.smithi078.zidbgq (mgr.14162) 315 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:10.277462+0000 mgr.smithi078.zidbgq (mgr.14162) 315 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:13.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:12.277925+0000 mgr.smithi078.zidbgq (mgr.14162) 316 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:12.277925+0000 mgr.smithi078.zidbgq (mgr.14162) 316 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:14.278374+0000 mgr.smithi078.zidbgq (mgr.14162) 317 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:14.278374+0000 mgr.smithi078.zidbgq (mgr.14162) 317 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:17.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:16.278784+0000 mgr.smithi078.zidbgq (mgr.14162) 318 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:17.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:16.278784+0000 mgr.smithi078.zidbgq (mgr.14162) 318 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:19.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:18.279252+0000 mgr.smithi078.zidbgq (mgr.14162) 319 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:19.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:18.279252+0000 mgr.smithi078.zidbgq (mgr.14162) 319 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:21.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:20.279689+0000 mgr.smithi078.zidbgq (mgr.14162) 320 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:21.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:21.070024+0000 mon.smithi078 (mon.0) 378 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:49:21.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:21.304668+0000 mon.smithi078 (mon.0) 379 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:21.798 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:20.279689+0000 mgr.smithi078.zidbgq (mgr.14162) 320 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:21.798 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:21.070024+0000 mon.smithi078 (mon.0) 378 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:49:21.798 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:21.304668+0000 mon.smithi078 (mon.0) 379 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:49:21.069378+0000 mgr.smithi078.zidbgq (mgr.14162) 321 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi148 ... 2023-12-21T03:49:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi148 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-12-21T03:49:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-12-21T03:49:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:47:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:48:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:48:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-21T03:49:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:49:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: main() 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: r = ctx.func(ctx) 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: return func(ctx) 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: raise RuntimeError('uid/gid not found') 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: RuntimeError: uid/gid not found 2023-12-21T03:49:22.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: yield (conn, connr) 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: code, '\n'.join(err))) 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi148 ... 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi148 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:47:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:48:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:48:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:49:22.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: main() 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: r = ctx.func(ctx) 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: return func(ctx) 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: raise RuntimeError('uid/gid not found') 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: RuntimeError: uid/gid not found 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:22.091948+0000 mon.smithi078 (mon.0) 380 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:22.092504+0000 mon.smithi078 (mon.0) 381 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:49:22.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:22.169323+0000 mon.smithi078 (mon.0) 382 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:22.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:49:21.069378+0000 mgr.smithi078.zidbgq (mgr.14162) 321 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi148 ... 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi148 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:47:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:48:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:48:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:49:22.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('uid/gid not found') 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: uid/gid not found 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: yield (conn, connr) 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: code, '\n'.join(err))) 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi148 ... 2023-12-21T03:49:22.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi148 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:47:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:48:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:48:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-21T03:49:22.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-21T03:49:22.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-12-21T03:49:22.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-12-21T03:49:22.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-21T03:49:22.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('uid/gid not found') 2023-12-21T03:49:22.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: uid/gid not found 2023-12-21T03:49:22.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:22.091948+0000 mon.smithi078 (mon.0) 380 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-21T03:49:22.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:22.092504+0000 mon.smithi078 (mon.0) 381 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:49:22.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:22.169323+0000 mon.smithi078 (mon.0) 382 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:23.627 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:49:22.092914+0000 mgr.smithi078.zidbgq (mgr.14162) 322 : cephadm [INF] Deploying daemon osd.3 on smithi148 2023-12-21T03:49:23.627 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:22.280009+0000 mgr.smithi078.zidbgq (mgr.14162) 323 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:23.627 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:23.019733+0000 mon.smithi078 (mon.0) 383 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:23.708 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:49:22.092914+0000 mgr.smithi078.zidbgq (mgr.14162) 322 : cephadm [INF] Deploying daemon osd.3 on smithi148 2023-12-21T03:49:23.709 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:22.280009+0000 mgr.smithi078.zidbgq (mgr.14162) 323 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:23.709 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:23.019733+0000 mon.smithi078 (mon.0) 383 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:24.408 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 3 on host 'smithi148' 2023-12-21T03:49:24.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.406+0000 7f3e277fe700 1 -- 172.21.15.148:0/2015495900 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3e40057f10 con 0x7f3e2805c2b0 2023-12-21T03:49:24.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.408+0000 7f3e45967700 1 -- 172.21.15.148:0/2015495900 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3e2805c2b0 msgr2=0x7f3e2805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:49:24.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.408+0000 7f3e45967700 1 --2- 172.21.15.148:0/2015495900 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3e2805c2b0 0x7f3e2805e770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f3e34009f50 tx=0x7f3e34009b30).stop 2023-12-21T03:49:24.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.408+0000 7f3e45967700 1 -- 172.21.15.148:0/2015495900 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e400f55c0 msgr2=0x7f3e40109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:49:24.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.408+0000 7f3e45967700 1 --2- 172.21.15.148:0/2015495900 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e400f55c0 0x7f3e40109cb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f3e30002700 tx=0x7f3e30009d80).stop 2023-12-21T03:49:24.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.408+0000 7f3e45967700 1 -- 172.21.15.148:0/2015495900 shutdown_connections 2023-12-21T03:49:24.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.408+0000 7f3e45967700 1 --2- 172.21.15.148:0/2015495900 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f3e2805c2b0 0x7f3e2805e770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:49:24.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.408+0000 7f3e45967700 1 --2- 172.21.15.148:0/2015495900 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e400f55c0 0x7f3e40109cb0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:49:24.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.408+0000 7f3e45967700 1 --2- 172.21.15.148:0/2015495900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e400f6860 0x7f3e4010c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:49:24.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.408+0000 7f3e45967700 1 -- 172.21.15.148:0/2015495900 >> 172.21.15.148:0/2015495900 conn(0x7f3e400f0d40 msgr2=0x7f3e400fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:49:24.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.408+0000 7f3e45967700 1 -- 172.21.15.148:0/2015495900 shutdown_connections 2023-12-21T03:49:24.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:24.408+0000 7f3e45967700 1 -- 172.21.15.148:0/2015495900 wait complete. 2023-12-21T03:49:24.773 DEBUG:teuthology.orchestra.run.smithi148:osd.3> sudo journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.3.service 2023-12-21T03:49:24.775 INFO:tasks.cephadm:Deploying osd.4 on smithi148 with /dev/vg_nvme/lv_3... 2023-12-21T03:49:24.776 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-21T03:49:24.805 INFO:journalctl@ceph.osd.3.smithi148.stdout:-- Logs begin at Thu 2023-12-21 03:26:40 UTC. -- 2023-12-21T03:49:25.131 INFO:journalctl@ceph.osd.3.smithi148.stdout:Dec 21 03:49:24 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.3[133928]: debug 2023-12-21T03:49:24.871+0000 7f22b9f36080 -1 Falling back to public interface 2023-12-21T03:49:25.745 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:24.280323+0000 mgr.smithi078.zidbgq (mgr.14162) 324 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:25.745 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:24.403420+0000 mon.smithi078 (mon.0) 384 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:25.745 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:24.406514+0000 mon.smithi078 (mon.0) 385 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:25.745 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:24.408377+0000 mon.smithi078 (mon.0) 386 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2023-12-21T03:49:25.745 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:25.354981+0000 mon.smithi078 (mon.0) 387 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:24.280323+0000 mgr.smithi078.zidbgq (mgr.14162) 324 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:24.403420+0000 mon.smithi078 (mon.0) 384 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:24.406514+0000 mon.smithi078 (mon.0) 385 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:24.408377+0000 mon.smithi078 (mon.0) 386 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2023-12-21T03:49:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:25.354981+0000 mon.smithi078 (mon.0) 387 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:26.198 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-21T03:49:26.198 INFO:teuthology.orchestra.run.smithi148.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-21T03:49:26.198 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10+0 records in 2023-12-21T03:49:26.198 INFO:teuthology.orchestra.run.smithi148.stderr:10+0 records out 2023-12-21T03:49:26.198 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122612 s, 855 MB/s 2023-12-21T03:49:26.198 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping successful for: 2023-12-21T03:49:26.515 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:49:25.350199+0000 mgr.smithi078.zidbgq (mgr.14162) 325 : cephadm [INF] Detected new or changed devices on smithi078 2023-12-21T03:49:26.515 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:25.414518+0000 mon.smithi078 (mon.0) 388 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2023-12-21T03:49:26.515 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:25.414584+0000 mon.smithi078 (mon.0) 389 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2023-12-21T03:49:26.515 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:25.414709+0000 mon.smithi078 (mon.0) 390 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:26.516 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:25.415398+0000 mon.smithi078 (mon.0) 391 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-12-21T03:49:26.516 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:25.608705+0000 mon.smithi078 (mon.0) 392 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:26.541 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi148:vg_nvme/lv_3 2023-12-21T03:49:26.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:49:25.350199+0000 mgr.smithi078.zidbgq (mgr.14162) 325 : cephadm [INF] Detected new or changed devices on smithi078 2023-12-21T03:49:26.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:25.414518+0000 mon.smithi078 (mon.0) 388 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2023-12-21T03:49:26.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:25.414584+0000 mon.smithi078 (mon.0) 389 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2023-12-21T03:49:26.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:25.414709+0000 mon.smithi078 (mon.0) 390 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:26.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:25.415398+0000 mon.smithi078 (mon.0) 391 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-12-21T03:49:26.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:25.608705+0000 mon.smithi078 (mon.0) 392 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:49:27.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.649+0000 7f58d098b700 1 -- 172.21.15.148:0/3467345156 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f58cc060310 msgr2=0x7f58cc0f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:49:27.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.649+0000 7f58d098b700 1 --2- 172.21.15.148:0/3467345156 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f58cc060310 0x7f58cc0f9110 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f58bc004660 tx=0x7f58bc009b30).stop 2023-12-21T03:49:27.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.649+0000 7f58d098b700 1 -- 172.21.15.148:0/3467345156 shutdown_connections 2023-12-21T03:49:27.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.649+0000 7f58d098b700 1 --2- 172.21.15.148:0/3467345156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58cc0f9690 0x7f58cc0fbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:49:27.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.649+0000 7f58d098b700 1 --2- 172.21.15.148:0/3467345156 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f58cc060310 0x7f58cc0f9110 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.649+0000 7f58d098b700 1 -- 172.21.15.148:0/3467345156 >> 172.21.15.148:0/3467345156 conn(0x7f58cc0f0b90 msgr2=0x7f58cc0f2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.649+0000 7f58d098b700 1 -- 172.21.15.148:0/3467345156 shutdown_connections 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.649+0000 7f58d098b700 1 -- 172.21.15.148:0/3467345156 wait complete. 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58d098b700 1 Processor -- start 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58d098b700 1 -- start start 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58d098b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58cc060310 0x7f58cc106d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58d098b700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f58cc0f9690 0x7f58cc107260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58d098b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc107860 con 0x7f58cc060310 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58d098b700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc108070 con 0x7f58cc0f9690 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58ca59c700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f58cc0f9690 0x7f58cc107260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58ca59c700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f58cc0f9690 0x7f58cc107260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52656/0 (socket says 172.21.15.148:52656) 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58ca59c700 1 -- 172.21.15.148:0/2989965372 learned_addr learned my addr 172.21.15.148:0/2989965372 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:49:27.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58cad9d700 1 --2- 172.21.15.148:0/2989965372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58cc060310 0x7f58cc106d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:49:27.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58ca59c700 1 -- 172.21.15.148:0/2989965372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58cc060310 msgr2=0x7f58cc106d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:49:27.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58ca59c700 1 --2- 172.21.15.148:0/2989965372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58cc060310 0x7f58cc106d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:49:27.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.650+0000 7f58ca59c700 1 -- 172.21.15.148:0/2989965372 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc005040 con 0x7f58cc0f9690 2023-12-21T03:49:27.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.651+0000 7f58ca59c700 1 --2- 172.21.15.148:0/2989965372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f58cc0f9690 0x7f58cc107260 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f58c000e7c0 tx=0x7f58c000b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:49:27.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.651+0000 7f58b37fe700 1 -- 172.21.15.148:0/2989965372 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58c00092e0 con 0x7f58cc0f9690 2023-12-21T03:49:27.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.651+0000 7f58d098b700 1 -- 172.21.15.148:0/2989965372 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58cc1082f0 con 0x7f58cc0f9690 2023-12-21T03:49:27.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.651+0000 7f58b37fe700 1 -- 172.21.15.148:0/2989965372 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f58c000f040 con 0x7f58cc0f9690 2023-12-21T03:49:27.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.651+0000 7f58d098b700 1 -- 172.21.15.148:0/2989965372 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58cc1088b0 con 0x7f58cc0f9690 2023-12-21T03:49:27.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.651+0000 7f58b37fe700 1 -- 172.21.15.148:0/2989965372 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58c0014740 con 0x7f58cc0f9690 2023-12-21T03:49:27.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.652+0000 7f58d098b700 1 -- 172.21.15.148:0/2989965372 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58cc059100 con 0x7f58cc0f9690 2023-12-21T03:49:27.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.652+0000 7f58b37fe700 1 -- 172.21.15.148:0/2989965372 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f58c00077b0 con 0x7f58cc0f9690 2023-12-21T03:49:27.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.652+0000 7f58b37fe700 1 --2- 172.21.15.148:0/2989965372 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f58b405c300 0x7f58b405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:49:27.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.652+0000 7f58b37fe700 1 -- 172.21.15.148:0/2989965372 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3261+0+0 (secure 0 0 0) 0x7f58c0075950 con 0x7f58cc0f9690 2023-12-21T03:49:27.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.653+0000 7f58cad9d700 1 --2- 172.21.15.148:0/2989965372 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f58b405c300 0x7f58b405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:49:27.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.653+0000 7f58cad9d700 1 --2- 172.21.15.148:0/2989965372 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f58b405c300 0x7f58b405e7c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f58bc013530 tx=0x7f58bc016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:49:27.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.656+0000 7f58b37fe700 1 -- 172.21.15.148:0/2989965372 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58c0048e60 con 0x7f58cc0f9690 2023-12-21T03:49:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:25.605553+0000 mgr.smithi078.zidbgq (mgr.14162) 326 : cluster [DBG] pgmap v307: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:49:25.612280+0000 mgr.smithi078.zidbgq (mgr.14162) 327 : cephadm [INF] Deploying daemon prometheus.smithi148 on smithi148 2023-12-21T03:49:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:26.412113+0000 mon.smithi078 (mon.0) 393 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-12-21T03:49:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:26.419145+0000 mon.smithi078 (mon.0) 394 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-12-21T03:49:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:26.419257+0000 mon.smithi078 (mon.0) 395 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2023-12-21T03:49:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:26.419656+0000 mon.smithi078 (mon.0) 396 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:25.605553+0000 mgr.smithi078.zidbgq (mgr.14162) 326 : cluster [DBG] pgmap v307: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:49:25.612280+0000 mgr.smithi078.zidbgq (mgr.14162) 327 : cephadm [INF] Deploying daemon prometheus.smithi148 on smithi148 2023-12-21T03:49:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:26.412113+0000 mon.smithi078 (mon.0) 393 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-12-21T03:49:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:26.419145+0000 mon.smithi078 (mon.0) 394 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-12-21T03:49:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:26.419257+0000 mon.smithi078 (mon.0) 395 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2023-12-21T03:49:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:26.419656+0000 mon.smithi078 (mon.0) 396 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:27.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:49:27.831+0000 7f58d098b700 1 -- 172.21.15.148:0/2989965372 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f58cc0fee50 con 0x7f58b405c300 2023-12-21T03:49:28.766 INFO:journalctl@ceph.osd.3.smithi148.stdout:Dec 21 03:49:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.3[133928]: debug 2023-12-21T03:49:28.374+0000 7f22b9f36080 -1 osd.3 0 log_to_monitors {default=true} 2023-12-21T03:49:28.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:27.423499+0000 mon.smithi078 (mon.0) 397 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2023-12-21T03:49:28.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:27.423969+0000 mon.smithi078 (mon.0) 398 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:28.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:27.833997+0000 mon.smithi078 (mon.0) 399 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:49:28.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:27.835714+0000 mon.smithi078 (mon.0) 400 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:49:28.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:27.836467+0000 mon.smithi078 (mon.0) 401 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:49:28.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:28.378509+0000 mon.smithi148 (mon.1) 6 : audit [INF] from='osd.3 [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-21T03:49:28.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:28.379319+0000 mon.smithi078 (mon.0) 402 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-21T03:49:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:27.423499+0000 mon.smithi078 (mon.0) 397 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2023-12-21T03:49:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:27.423969+0000 mon.smithi078 (mon.0) 398 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:27.833997+0000 mon.smithi078 (mon.0) 399 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:49:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:27.835714+0000 mon.smithi078 (mon.0) 400 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:49:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:27.836467+0000 mon.smithi078 (mon.0) 401 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:49:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:28.378509+0000 mon.smithi148 (mon.1) 6 : audit [INF] from='osd.3 [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-21T03:49:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:28.379319+0000 mon.smithi078 (mon.0) 402 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-21T03:49:29.765 INFO:journalctl@ceph.osd.3.smithi148.stdout:Dec 21 03:49:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.3[133928]: debug 2023-12-21T03:49:29.440+0000 7f22b04a7700 -1 osd.3 0 waiting for initial osdmap 2023-12-21T03:49:29.765 INFO:journalctl@ceph.osd.3.smithi148.stdout:Dec 21 03:49:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.3[133928]: debug 2023-12-21T03:49:29.448+0000 7f22aa41e700 -1 osd.3 24 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2023-12-21T03:49:29.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:27.605957+0000 mgr.smithi078.zidbgq (mgr.14162) 328 : cluster [DBG] pgmap v310: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:29.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:27.833103+0000 mgr.smithi078.zidbgq (mgr.14162) 329 : audit [DBG] from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:49:29.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:49:27.835421+0000 mgr.smithi078.zidbgq (mgr.14162) 330 : cephadm [INF] Applying service osd.None on host smithi148... 2023-12-21T03:49:29.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:28.434218+0000 mon.smithi078 (mon.0) 403 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-21T03:49:29.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:28.434283+0000 mon.smithi078 (mon.0) 404 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2023-12-21T03:49:29.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:28.434580+0000 mon.smithi078 (mon.0) 405 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:29.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:28.438282+0000 mon.smithi148 (mon.1) 7 : audit [INF] from='osd.3 [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:49:29.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:28.438917+0000 mon.smithi078 (mon.0) 406 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:49:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:27.605957+0000 mgr.smithi078.zidbgq (mgr.14162) 328 : cluster [DBG] pgmap v310: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:27.833103+0000 mgr.smithi078.zidbgq (mgr.14162) 329 : audit [DBG] from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:49:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:49:27.835421+0000 mgr.smithi078.zidbgq (mgr.14162) 330 : cephadm [INF] Applying service osd.None on host smithi148... 2023-12-21T03:49:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:28.434218+0000 mon.smithi078 (mon.0) 403 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-21T03:49:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:28.434283+0000 mon.smithi078 (mon.0) 404 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2023-12-21T03:49:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:28.434580+0000 mon.smithi078 (mon.0) 405 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:29.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:28.438282+0000 mon.smithi148 (mon.1) 7 : audit [INF] from='osd.3 [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:49:29.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:28.438917+0000 mon.smithi078 (mon.0) 406 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:49:30.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:29.437489+0000 mon.smithi078 (mon.0) 407 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2023-12-21T03:49:30.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:29.437556+0000 mon.smithi078 (mon.0) 408 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2023-12-21T03:49:30.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:29.438258+0000 mon.smithi078 (mon.0) 409 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:30.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:29.444039+0000 mon.smithi078 (mon.0) 410 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:30.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:29.437489+0000 mon.smithi078 (mon.0) 407 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2023-12-21T03:49:30.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:29.437556+0000 mon.smithi078 (mon.0) 408 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2023-12-21T03:49:30.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:29.438258+0000 mon.smithi078 (mon.0) 409 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:30.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:29.444039+0000 mon.smithi078 (mon.0) 410 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:29.356217+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:49:31.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:29.356338+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:49:31.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:29.606446+0000 mgr.smithi078.zidbgq (mgr.14162) 331 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:31.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:30.441781+0000 mon.smithi078 (mon.0) 411 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:31.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:30.449047+0000 mon.smithi078 (mon.0) 412 : cluster [INF] osd.3 [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634] boot 2023-12-21T03:49:31.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:30.449107+0000 mon.smithi078 (mon.0) 413 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2023-12-21T03:49:31.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:30.449283+0000 mon.smithi078 (mon.0) 414 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:31.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:30.558188+0000 mon.smithi078 (mon.0) 415 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:49:31.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:49:30.558369+0000 mon.smithi078 (mon.0) 416 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:49:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:29.356217+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:49:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:29.356338+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:49:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:29.606446+0000 mgr.smithi078.zidbgq (mgr.14162) 331 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-12-21T03:49:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:30.441781+0000 mon.smithi078 (mon.0) 411 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:30.449047+0000 mon.smithi078 (mon.0) 412 : cluster [INF] osd.3 [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634] boot 2023-12-21T03:49:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:30.449107+0000 mon.smithi078 (mon.0) 413 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2023-12-21T03:49:31.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:30.449283+0000 mon.smithi078 (mon.0) 414 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-21T03:49:31.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:30.558188+0000 mon.smithi078 (mon.0) 415 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:49:31.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:49:30.558369+0000 mon.smithi078 (mon.0) 416 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:49:32.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:32 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:31.454786+0000 mon.smithi078 (mon.0) 417 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2023-12-21T03:49:32.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:31.454786+0000 mon.smithi078 (mon.0) 417 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2023-12-21T03:49:33.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:31.606878+0000 mgr.smithi078.zidbgq (mgr.14162) 332 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 0 B data, 19 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:33.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:32.459399+0000 mon.smithi078 (mon.0) 418 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2023-12-21T03:49:33.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:31.606878+0000 mgr.smithi078.zidbgq (mgr.14162) 332 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 0 B data, 19 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:33.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:32.459399+0000 mon.smithi078 (mon.0) 418 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2023-12-21T03:49:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:33.607421+0000 mgr.smithi078.zidbgq (mgr.14162) 333 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:35.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:33.607421+0000 mgr.smithi078.zidbgq (mgr.14162) 333 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:37.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:35.607794+0000 mgr.smithi078.zidbgq (mgr.14162) 334 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:35.607794+0000 mgr.smithi078.zidbgq (mgr.14162) 334 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:39.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:37.608307+0000 mgr.smithi078.zidbgq (mgr.14162) 335 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:39.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:37.608307+0000 mgr.smithi078.zidbgq (mgr.14162) 335 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:41.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:39.608673+0000 mgr.smithi078.zidbgq (mgr.14162) 336 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:39.608673+0000 mgr.smithi078.zidbgq (mgr.14162) 336 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:41.608996+0000 mgr.smithi078.zidbgq (mgr.14162) 337 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:41.608996+0000 mgr.smithi078.zidbgq (mgr.14162) 337 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:43.609409+0000 mgr.smithi078.zidbgq (mgr.14162) 338 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:43.609409+0000 mgr.smithi078.zidbgq (mgr.14162) 338 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:45.609774+0000 mgr.smithi078.zidbgq (mgr.14162) 339 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:45.609774+0000 mgr.smithi078.zidbgq (mgr.14162) 339 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:47.610322+0000 mgr.smithi078.zidbgq (mgr.14162) 340 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:49.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:47.610322+0000 mgr.smithi078.zidbgq (mgr.14162) 340 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:49.610840+0000 mgr.smithi078.zidbgq (mgr.14162) 341 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:49.610840+0000 mgr.smithi078.zidbgq (mgr.14162) 341 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:53.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:51.611300+0000 mgr.smithi078.zidbgq (mgr.14162) 342 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:51.611300+0000 mgr.smithi078.zidbgq (mgr.14162) 342 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:53.611651+0000 mgr.smithi078.zidbgq (mgr.14162) 343 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:53.611651+0000 mgr.smithi078.zidbgq (mgr.14162) 343 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:55.612031+0000 mgr.smithi078.zidbgq (mgr.14162) 344 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:55.612031+0000 mgr.smithi078.zidbgq (mgr.14162) 344 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:49:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:57.612497+0000 mgr.smithi078.zidbgq (mgr.14162) 345 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:49:59.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:49:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:57.612497+0000 mgr.smithi078.zidbgq (mgr.14162) 345 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:00.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:00.000173+0000 mon.smithi078 (mon.0) 419 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-12-21T03:50:00.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:00.000242+0000 mon.smithi078 (mon.0) 420 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-12-21T03:50:00.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:00.000255+0000 mon.smithi078 (mon.0) 421 : cluster [WRN] daemon node-exporter.smithi078 on smithi078 is in unknown state 2023-12-21T03:50:00.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:00.000269+0000 mon.smithi078 (mon.0) 422 : cluster [WRN] daemon node-exporter.smithi148 on smithi148 is in unknown state 2023-12-21T03:50:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:00.000173+0000 mon.smithi078 (mon.0) 419 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-12-21T03:50:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:00.000242+0000 mon.smithi078 (mon.0) 420 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-12-21T03:50:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:00.000255+0000 mon.smithi078 (mon.0) 421 : cluster [WRN] daemon node-exporter.smithi078 on smithi078 is in unknown state 2023-12-21T03:50:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:00.000269+0000 mon.smithi078 (mon.0) 422 : cluster [WRN] daemon node-exporter.smithi148 on smithi148 is in unknown state 2023-12-21T03:50:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:49:59.613044+0000 mgr.smithi078.zidbgq (mgr.14162) 346 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:49:59.613044+0000 mgr.smithi078.zidbgq (mgr.14162) 346 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:01.613462+0000 mgr.smithi078.zidbgq (mgr.14162) 347 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:04.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:01.613462+0000 mgr.smithi078.zidbgq (mgr.14162) 347 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:05.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:03.613836+0000 mgr.smithi078.zidbgq (mgr.14162) 348 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:06.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:03.613836+0000 mgr.smithi078.zidbgq (mgr.14162) 348 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:05.614274+0000 mgr.smithi078.zidbgq (mgr.14162) 349 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:08.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:05.614274+0000 mgr.smithi078.zidbgq (mgr.14162) 349 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:09.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:07.614752+0000 mgr.smithi078.zidbgq (mgr.14162) 350 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:10.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:07.614752+0000 mgr.smithi078.zidbgq (mgr.14162) 350 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:09.615164+0000 mgr.smithi078.zidbgq (mgr.14162) 351 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:12.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:09.615164+0000 mgr.smithi078.zidbgq (mgr.14162) 351 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:11.615574+0000 mgr.smithi078.zidbgq (mgr.14162) 352 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:14.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:11.615574+0000 mgr.smithi078.zidbgq (mgr.14162) 352 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:13.615923+0000 mgr.smithi078.zidbgq (mgr.14162) 353 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:16.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:13.615923+0000 mgr.smithi078.zidbgq (mgr.14162) 353 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:17.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:15.616407+0000 mgr.smithi078.zidbgq (mgr.14162) 354 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:18.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:15.616407+0000 mgr.smithi078.zidbgq (mgr.14162) 354 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:19.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:17.616910+0000 mgr.smithi078.zidbgq (mgr.14162) 355 : cluster [DBG] pgmap v340: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:20.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:17.616910+0000 mgr.smithi078.zidbgq (mgr.14162) 355 : cluster [DBG] pgmap v340: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:19.617387+0000 mgr.smithi078.zidbgq (mgr.14162) 356 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:22.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:19.617387+0000 mgr.smithi078.zidbgq (mgr.14162) 356 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:21.617712+0000 mgr.smithi078.zidbgq (mgr.14162) 357 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:23.929 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:21.617712+0000 mgr.smithi078.zidbgq (mgr.14162) 357 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:23.618064+0000 mgr.smithi078.zidbgq (mgr.14162) 358 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:23.618064+0000 mgr.smithi078.zidbgq (mgr.14162) 358 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:25.618419+0000 mgr.smithi078.zidbgq (mgr.14162) 359 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:25.618419+0000 mgr.smithi078.zidbgq (mgr.14162) 359 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:29.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:27.618754+0000 mgr.smithi078.zidbgq (mgr.14162) 360 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:27.618754+0000 mgr.smithi078.zidbgq (mgr.14162) 360 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:29.619074+0000 mgr.smithi078.zidbgq (mgr.14162) 361 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:50:30.560694+0000 mon.smithi078 (mon.0) 423 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:50:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:50:30.560876+0000 mon.smithi078 (mon.0) 424 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:50:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:29.619074+0000 mgr.smithi078.zidbgq (mgr.14162) 361 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:50:30.560694+0000 mon.smithi078 (mon.0) 423 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:50:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:50:30.560876+0000 mon.smithi078 (mon.0) 424 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:50:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:31.619541+0000 mgr.smithi078.zidbgq (mgr.14162) 362 : cluster [DBG] pgmap v347: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:33.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:31.619541+0000 mgr.smithi078.zidbgq (mgr.14162) 362 : cluster [DBG] pgmap v347: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:33.620033+0000 mgr.smithi078.zidbgq (mgr.14162) 363 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:35.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:33.620033+0000 mgr.smithi078.zidbgq (mgr.14162) 363 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:37.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:35.620362+0000 mgr.smithi078.zidbgq (mgr.14162) 364 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:35.620362+0000 mgr.smithi078.zidbgq (mgr.14162) 364 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:39.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:37.620883+0000 mgr.smithi078.zidbgq (mgr.14162) 365 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:39.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:37.620883+0000 mgr.smithi078.zidbgq (mgr.14162) 365 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:41.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:39.621295+0000 mgr.smithi078.zidbgq (mgr.14162) 366 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:39.621295+0000 mgr.smithi078.zidbgq (mgr.14162) 366 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:41.621576+0000 mgr.smithi078.zidbgq (mgr.14162) 367 : cluster [DBG] pgmap v352: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:41.621576+0000 mgr.smithi078.zidbgq (mgr.14162) 367 : cluster [DBG] pgmap v352: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:43.622050+0000 mgr.smithi078.zidbgq (mgr.14162) 368 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:43.622050+0000 mgr.smithi078.zidbgq (mgr.14162) 368 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:45.622419+0000 mgr.smithi078.zidbgq (mgr.14162) 369 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:45.622419+0000 mgr.smithi078.zidbgq (mgr.14162) 369 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:47.622903+0000 mgr.smithi078.zidbgq (mgr.14162) 370 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:49.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:47.622903+0000 mgr.smithi078.zidbgq (mgr.14162) 370 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:49.623357+0000 mgr.smithi078.zidbgq (mgr.14162) 371 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:49.623357+0000 mgr.smithi078.zidbgq (mgr.14162) 371 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:53.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:51.623779+0000 mgr.smithi078.zidbgq (mgr.14162) 372 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:51.623779+0000 mgr.smithi078.zidbgq (mgr.14162) 372 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:53.624245+0000 mgr.smithi078.zidbgq (mgr.14162) 373 : cluster [DBG] pgmap v358: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:55.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:53.624245+0000 mgr.smithi078.zidbgq (mgr.14162) 373 : cluster [DBG] pgmap v358: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:55.624664+0000 mgr.smithi078.zidbgq (mgr.14162) 374 : cluster [DBG] pgmap v359: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:55.624664+0000 mgr.smithi078.zidbgq (mgr.14162) 374 : cluster [DBG] pgmap v359: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:50:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:57.625179+0000 mgr.smithi078.zidbgq (mgr.14162) 375 : cluster [DBG] pgmap v360: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:50:59.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:50:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:57.625179+0000 mgr.smithi078.zidbgq (mgr.14162) 375 : cluster [DBG] pgmap v360: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:50:59.625623+0000 mgr.smithi078.zidbgq (mgr.14162) 376 : cluster [DBG] pgmap v361: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:50:59.625623+0000 mgr.smithi078.zidbgq (mgr.14162) 376 : cluster [DBG] pgmap v361: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:03.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:01.625951+0000 mgr.smithi078.zidbgq (mgr.14162) 377 : cluster [DBG] pgmap v362: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:01.625951+0000 mgr.smithi078.zidbgq (mgr.14162) 377 : cluster [DBG] pgmap v362: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:03.626474+0000 mgr.smithi078.zidbgq (mgr.14162) 378 : cluster [DBG] pgmap v363: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:05.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:03.626474+0000 mgr.smithi078.zidbgq (mgr.14162) 378 : cluster [DBG] pgmap v363: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:07.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:05.626929+0000 mgr.smithi078.zidbgq (mgr.14162) 379 : cluster [DBG] pgmap v364: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:05.626929+0000 mgr.smithi078.zidbgq (mgr.14162) 379 : cluster [DBG] pgmap v364: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:09.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:07.627422+0000 mgr.smithi078.zidbgq (mgr.14162) 380 : cluster [DBG] pgmap v365: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:09.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:07.627422+0000 mgr.smithi078.zidbgq (mgr.14162) 380 : cluster [DBG] pgmap v365: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:09.627860+0000 mgr.smithi078.zidbgq (mgr.14162) 381 : cluster [DBG] pgmap v366: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:09.627860+0000 mgr.smithi078.zidbgq (mgr.14162) 381 : cluster [DBG] pgmap v366: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:13.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:11.628206+0000 mgr.smithi078.zidbgq (mgr.14162) 382 : cluster [DBG] pgmap v367: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:11.628206+0000 mgr.smithi078.zidbgq (mgr.14162) 382 : cluster [DBG] pgmap v367: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:13.628631+0000 mgr.smithi078.zidbgq (mgr.14162) 383 : cluster [DBG] pgmap v368: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:13.628631+0000 mgr.smithi078.zidbgq (mgr.14162) 383 : cluster [DBG] pgmap v368: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:17.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:15.628983+0000 mgr.smithi078.zidbgq (mgr.14162) 384 : cluster [DBG] pgmap v369: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:17.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:15.628983+0000 mgr.smithi078.zidbgq (mgr.14162) 384 : cluster [DBG] pgmap v369: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:19.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:17.629479+0000 mgr.smithi078.zidbgq (mgr.14162) 385 : cluster [DBG] pgmap v370: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:19.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:17.629479+0000 mgr.smithi078.zidbgq (mgr.14162) 385 : cluster [DBG] pgmap v370: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:21.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:19.629964+0000 mgr.smithi078.zidbgq (mgr.14162) 386 : cluster [DBG] pgmap v371: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:19.629964+0000 mgr.smithi078.zidbgq (mgr.14162) 386 : cluster [DBG] pgmap v371: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:23.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:21.630449+0000 mgr.smithi078.zidbgq (mgr.14162) 387 : cluster [DBG] pgmap v372: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:21.630449+0000 mgr.smithi078.zidbgq (mgr.14162) 387 : cluster [DBG] pgmap v372: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:23.630901+0000 mgr.smithi078.zidbgq (mgr.14162) 388 : cluster [DBG] pgmap v373: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:23.630901+0000 mgr.smithi078.zidbgq (mgr.14162) 388 : cluster [DBG] pgmap v373: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:25.631414+0000 mgr.smithi078.zidbgq (mgr.14162) 389 : cluster [DBG] pgmap v374: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:25.631414+0000 mgr.smithi078.zidbgq (mgr.14162) 389 : cluster [DBG] pgmap v374: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:29.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:27.631859+0000 mgr.smithi078.zidbgq (mgr.14162) 390 : cluster [DBG] pgmap v375: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:27.631859+0000 mgr.smithi078.zidbgq (mgr.14162) 390 : cluster [DBG] pgmap v375: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:29.632380+0000 mgr.smithi078.zidbgq (mgr.14162) 391 : cluster [DBG] pgmap v376: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:30.563110+0000 mon.smithi078 (mon.0) 425 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:51:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:30.563348+0000 mon.smithi078 (mon.0) 426 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:51:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:29.632380+0000 mgr.smithi078.zidbgq (mgr.14162) 391 : cluster [DBG] pgmap v376: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:30.563110+0000 mon.smithi078 (mon.0) 425 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:51:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:30.563348+0000 mon.smithi078 (mon.0) 426 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:51:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:31.632891+0000 mgr.smithi078.zidbgq (mgr.14162) 392 : cluster [DBG] pgmap v377: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:33.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:31.632891+0000 mgr.smithi078.zidbgq (mgr.14162) 392 : cluster [DBG] pgmap v377: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:33.633276+0000 mgr.smithi078.zidbgq (mgr.14162) 393 : cluster [DBG] pgmap v378: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:35.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:33.633276+0000 mgr.smithi078.zidbgq (mgr.14162) 393 : cluster [DBG] pgmap v378: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:37.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:35.633551+0000 mgr.smithi078.zidbgq (mgr.14162) 394 : cluster [DBG] pgmap v379: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:35.633551+0000 mgr.smithi078.zidbgq (mgr.14162) 394 : cluster [DBG] pgmap v379: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:39.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:37.633907+0000 mgr.smithi078.zidbgq (mgr.14162) 395 : cluster [DBG] pgmap v380: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:39.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:37.633907+0000 mgr.smithi078.zidbgq (mgr.14162) 395 : cluster [DBG] pgmap v380: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:41.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:39.634323+0000 mgr.smithi078.zidbgq (mgr.14162) 396 : cluster [DBG] pgmap v381: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:39.634323+0000 mgr.smithi078.zidbgq (mgr.14162) 396 : cluster [DBG] pgmap v381: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:41.634705+0000 mgr.smithi078.zidbgq (mgr.14162) 397 : cluster [DBG] pgmap v382: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:41.634705+0000 mgr.smithi078.zidbgq (mgr.14162) 397 : cluster [DBG] pgmap v382: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:43.635050+0000 mgr.smithi078.zidbgq (mgr.14162) 398 : cluster [DBG] pgmap v383: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:43.635050+0000 mgr.smithi078.zidbgq (mgr.14162) 398 : cluster [DBG] pgmap v383: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:45.635415+0000 mgr.smithi078.zidbgq (mgr.14162) 399 : cluster [DBG] pgmap v384: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:45.635415+0000 mgr.smithi078.zidbgq (mgr.14162) 399 : cluster [DBG] pgmap v384: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:47.635896+0000 mgr.smithi078.zidbgq (mgr.14162) 400 : cluster [DBG] pgmap v385: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:49.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:47.635896+0000 mgr.smithi078.zidbgq (mgr.14162) 400 : cluster [DBG] pgmap v385: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:49.636263+0000 mgr.smithi078.zidbgq (mgr.14162) 401 : cluster [DBG] pgmap v386: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:49.636263+0000 mgr.smithi078.zidbgq (mgr.14162) 401 : cluster [DBG] pgmap v386: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:53.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:51.636642+0000 mgr.smithi078.zidbgq (mgr.14162) 402 : cluster [DBG] pgmap v387: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:51.636642+0000 mgr.smithi078.zidbgq (mgr.14162) 402 : cluster [DBG] pgmap v387: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:53.637159+0000 mgr.smithi078.zidbgq (mgr.14162) 403 : cluster [DBG] pgmap v388: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:55.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:54.758581+0000 mon.smithi078 (mon.0) 427 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-21T03:51:55.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:54.759453+0000 mon.smithi078 (mon.0) 428 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-21T03:51:55.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:54.760245+0000 mon.smithi078 (mon.0) 429 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:51:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:53.637159+0000 mgr.smithi078.zidbgq (mgr.14162) 403 : cluster [DBG] pgmap v388: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:54.758581+0000 mon.smithi078 (mon.0) 427 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-21T03:51:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:54.759453+0000 mon.smithi078 (mon.0) 428 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-21T03:51:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:54.760245+0000 mon.smithi078 (mon.0) 429 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:51:56.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:51:54.757783+0000 mgr.smithi078.zidbgq (mgr.14162) 404 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi148 ... 2023-12-21T03:51:56.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi148 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-12-21T03:51:56.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-12-21T03:51:56.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:50:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:50:39Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:51:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: main() 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: r = ctx.func(ctx) 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: return func(ctx) 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-12-21T03:51:56.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: raise RuntimeError('uid/gid not found') 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: RuntimeError: uid/gid not found 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: yield (conn, connr) 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: code, '\n'.join(err))) 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi148 ... 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi148 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:50:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.768 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:50:39Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr time="2023-12-21T03:51:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: main() 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: r = ctx.func(ctx) 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: return func(ctx) 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-12-21T03:51:56.769 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-12-21T03:51:56.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-21T03:51:56.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: raise RuntimeError('uid/gid not found') 2023-12-21T03:51:56.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: RuntimeError: uid/gid not found 2023-12-21T03:51:56.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:51:54.758233+0000 mgr.smithi078.zidbgq (mgr.14162) 405 : cephadm [INF] Reconfiguring mon.smithi078 (unknown last config time)... 2023-12-21T03:51:56.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:51:54.760832+0000 mgr.smithi078.zidbgq (mgr.14162) 406 : cephadm [INF] Reconfiguring daemon mon.smithi078 on smithi078 2023-12-21T03:51:56.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:55.577633+0000 mon.smithi148 (mon.1) 8 : audit [INF] from='client.? 172.21.15.148:0/2390272760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdb70954-ff3d-4fe1-a0e2-d3c162a98a7a"}]: dispatch 2023-12-21T03:51:56.770 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:55.577976+0000 mon.smithi078 (mon.0) 430 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdb70954-ff3d-4fe1-a0e2-d3c162a98a7a"}]: dispatch 2023-12-21T03:51:56.771 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:55.582467+0000 mon.smithi078 (mon.0) 431 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cdb70954-ff3d-4fe1-a0e2-d3c162a98a7a"}]': finished 2023-12-21T03:51:56.771 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:55.582534+0000 mon.smithi078 (mon.0) 432 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2023-12-21T03:51:56.771 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:55.583217+0000 mon.smithi078 (mon.0) 433 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-21T03:51:56.771 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:55.627189+0000 mon.smithi078 (mon.0) 434 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:51:56.771 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:55.627953+0000 mon.smithi078 (mon.0) 435 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi078.zidbgq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-21T03:51:56.771 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:55.628515+0000 mon.smithi078 (mon.0) 436 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-21T03:51:56.771 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:55.628957+0000 mon.smithi078 (mon.0) 437 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:51:56.771 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:55.990653+0000 mon.smithi148 (mon.1) 9 : audit [DBG] from='client.? 172.21.15.148:0/1572074810' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:51:56.771 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:56.483928+0000 mon.smithi078 (mon.0) 438 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:51:56.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:51:54.757783+0000 mgr.smithi078.zidbgq (mgr.14162) 404 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi148 ... 2023-12-21T03:51:56.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi148 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-12-21T03:51:56.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-12-21T03:51:56.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:50:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:50:39Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:51:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-21T03:51:56.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:51:56.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:51:56.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:51:56.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:51:56.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:51:56.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:51:56.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:51:56.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-21T03:51:56.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-21T03:51:56.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-12-21T03:51:56.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-12-21T03:51:56.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-21T03:51:56.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('uid/gid not found') 2023-12-21T03:51:56.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: uid/gid not found 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: yield (conn, connr) 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: code, '\n'.join(err))) 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi148 ... 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi148 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:50:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:50:39Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr time="2023-12-21T03:51:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-21T03:51:56.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:51:56.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:51:56.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:51:56.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:51:56.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:51:56.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:51:56.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:51:56.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-21T03:51:56.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-21T03:51:56.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-12-21T03:51:56.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-12-21T03:51:56.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('uid/gid not found') 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: uid/gid not found 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:51:54.758233+0000 mgr.smithi078.zidbgq (mgr.14162) 405 : cephadm [INF] Reconfiguring mon.smithi078 (unknown last config time)... 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:51:54.760832+0000 mgr.smithi078.zidbgq (mgr.14162) 406 : cephadm [INF] Reconfiguring daemon mon.smithi078 on smithi078 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:55.577633+0000 mon.smithi148 (mon.1) 8 : audit [INF] from='client.? 172.21.15.148:0/2390272760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdb70954-ff3d-4fe1-a0e2-d3c162a98a7a"}]: dispatch 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:55.577976+0000 mon.smithi078 (mon.0) 430 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdb70954-ff3d-4fe1-a0e2-d3c162a98a7a"}]: dispatch 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:55.582467+0000 mon.smithi078 (mon.0) 431 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cdb70954-ff3d-4fe1-a0e2-d3c162a98a7a"}]': finished 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:55.582534+0000 mon.smithi078 (mon.0) 432 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:55.583217+0000 mon.smithi078 (mon.0) 433 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:55.627189+0000 mon.smithi078 (mon.0) 434 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:55.627953+0000 mon.smithi078 (mon.0) 435 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi078.zidbgq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:55.628515+0000 mon.smithi078 (mon.0) 436 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-21T03:51:56.821 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:55.628957+0000 mon.smithi078 (mon.0) 437 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:51:56.822 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:55.990653+0000 mon.smithi148 (mon.1) 9 : audit [DBG] from='client.? 172.21.15.148:0/1572074810' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:51:56.822 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:56.483928+0000 mon.smithi078 (mon.0) 438 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:51:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:51:55.627716+0000 mgr.smithi078.zidbgq (mgr.14162) 407 : cephadm [INF] Reconfiguring mgr.smithi078.zidbgq (unknown last config time)... 2023-12-21T03:51:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:51:55.629366+0000 mgr.smithi078.zidbgq (mgr.14162) 408 : cephadm [INF] Reconfiguring daemon mgr.smithi078.zidbgq on smithi078 2023-12-21T03:51:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:55.637412+0000 mgr.smithi078.zidbgq (mgr.14162) 409 : cluster [DBG] pgmap v390: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:51:56.484524+0000 mgr.smithi078.zidbgq (mgr.14162) 410 : cephadm [INF] Reconfiguring alertmanager.smithi078 (dependencies changed)... 2023-12-21T03:51:57.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:51:56.486442+0000 mgr.smithi078.zidbgq (mgr.14162) 411 : cephadm [INF] Reconfiguring daemon alertmanager.smithi078 on smithi078 2023-12-21T03:51:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:51:55.627716+0000 mgr.smithi078.zidbgq (mgr.14162) 407 : cephadm [INF] Reconfiguring mgr.smithi078.zidbgq (unknown last config time)... 2023-12-21T03:51:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:51:55.629366+0000 mgr.smithi078.zidbgq (mgr.14162) 408 : cephadm [INF] Reconfiguring daemon mgr.smithi078.zidbgq on smithi078 2023-12-21T03:51:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:55.637412+0000 mgr.smithi078.zidbgq (mgr.14162) 409 : cluster [DBG] pgmap v390: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:51:56.484524+0000 mgr.smithi078.zidbgq (mgr.14162) 410 : cephadm [INF] Reconfiguring alertmanager.smithi078 (dependencies changed)... 2023-12-21T03:51:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:51:56.486442+0000 mgr.smithi078.zidbgq (mgr.14162) 411 : cephadm [INF] Reconfiguring daemon alertmanager.smithi078 on smithi078 2023-12-21T03:51:59.739 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:57.637711+0000 mgr.smithi078.zidbgq (mgr.14162) 412 : cluster [DBG] pgmap v391: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:59.739 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:59.099030+0000 mon.smithi078 (mon.0) 439 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:51:59.739 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:59.099831+0000 mon.smithi078 (mon.0) 440 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi078", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-21T03:51:59.740 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:51:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:59.100472+0000 mon.smithi078 (mon.0) 441 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:51:59.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:57.637711+0000 mgr.smithi078.zidbgq (mgr.14162) 412 : cluster [DBG] pgmap v391: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:51:59.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:59.099030+0000 mon.smithi078 (mon.0) 439 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:51:59.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:59.099831+0000 mon.smithi078 (mon.0) 440 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi078", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-21T03:51:59.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:51:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:59.100472+0000 mon.smithi078 (mon.0) 441 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:52:00.570 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:51:59.099572+0000 mgr.smithi078.zidbgq (mgr.14162) 413 : cephadm [INF] Reconfiguring crash.smithi078 (monmap changed)... 2023-12-21T03:52:00.570 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:51:59.100879+0000 mgr.smithi078.zidbgq (mgr.14162) 414 : cephadm [INF] Reconfiguring daemon crash.smithi078 on smithi078 2023-12-21T03:52:00.570 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:51:59.932791+0000 mon.smithi078 (mon.0) 442 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:52:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:51:59.099572+0000 mgr.smithi078.zidbgq (mgr.14162) 413 : cephadm [INF] Reconfiguring crash.smithi078 (monmap changed)... 2023-12-21T03:52:00.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:51:59.100879+0000 mgr.smithi078.zidbgq (mgr.14162) 414 : cephadm [INF] Reconfiguring daemon crash.smithi078 on smithi078 2023-12-21T03:52:00.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:51:59.932791+0000 mon.smithi078 (mon.0) 442 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:52:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:51:59.638025+0000 mgr.smithi078.zidbgq (mgr.14162) 415 : cluster [DBG] pgmap v392: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:51:59.933373+0000 mgr.smithi078.zidbgq (mgr.14162) 416 : cephadm [INF] Reconfiguring node-exporter.smithi078 (unknown last config time)... 2023-12-21T03:52:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:51:59.933486+0000 mgr.smithi078.zidbgq (mgr.14162) 417 : cephadm [INF] Reconfiguring daemon node-exporter.smithi078 on smithi078 2023-12-21T03:52:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:01.283720+0000 mon.smithi078 (mon.0) 443 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-21T03:52:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:01.284442+0000 mon.smithi078 (mon.0) 444 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:52:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:51:59.638025+0000 mgr.smithi078.zidbgq (mgr.14162) 415 : cluster [DBG] pgmap v392: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:51:59.933373+0000 mgr.smithi078.zidbgq (mgr.14162) 416 : cephadm [INF] Reconfiguring node-exporter.smithi078 (unknown last config time)... 2023-12-21T03:52:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:51:59.933486+0000 mgr.smithi078.zidbgq (mgr.14162) 417 : cephadm [INF] Reconfiguring daemon node-exporter.smithi078 on smithi078 2023-12-21T03:52:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:01.283720+0000 mon.smithi078 (mon.0) 443 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-21T03:52:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:01.284442+0000 mon.smithi078 (mon.0) 444 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:52:02.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:52:01.285026+0000 mgr.smithi078.zidbgq (mgr.14162) 418 : cephadm [INF] Deploying daemon osd.4 on smithi148 2023-12-21T03:52:02.898 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:02 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:52:01.285026+0000 mgr.smithi078.zidbgq (mgr.14162) 418 : cephadm [INF] Deploying daemon osd.4 on smithi148 2023-12-21T03:52:03.660 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 4 on host 'smithi148' 2023-12-21T03:52:03.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.657+0000 7f58b37fe700 1 -- 172.21.15.148:0/2989965372 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f58cc0fee50 con 0x7f58b405c300 2023-12-21T03:52:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.659+0000 7f58d098b700 1 -- 172.21.15.148:0/2989965372 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f58b405c300 msgr2=0x7f58b405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.659+0000 7f58d098b700 1 --2- 172.21.15.148:0/2989965372 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f58b405c300 0x7f58b405e7c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f58bc013530 tx=0x7f58bc016040).stop 2023-12-21T03:52:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.659+0000 7f58d098b700 1 -- 172.21.15.148:0/2989965372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f58cc0f9690 msgr2=0x7f58cc107260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.659+0000 7f58d098b700 1 --2- 172.21.15.148:0/2989965372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f58cc0f9690 0x7f58cc107260 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f58c000e7c0 tx=0x7f58c000b040).stop 2023-12-21T03:52:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.659+0000 7f58d098b700 1 -- 172.21.15.148:0/2989965372 shutdown_connections 2023-12-21T03:52:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.659+0000 7f58d098b700 1 --2- 172.21.15.148:0/2989965372 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f58b405c300 0x7f58b405e7c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:03.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.659+0000 7f58d098b700 1 --2- 172.21.15.148:0/2989965372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58cc060310 0x7f58cc106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:03.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.659+0000 7f58d098b700 1 --2- 172.21.15.148:0/2989965372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f58cc0f9690 0x7f58cc107260 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:03.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.659+0000 7f58d098b700 1 -- 172.21.15.148:0/2989965372 >> 172.21.15.148:0/2989965372 conn(0x7f58cc0f0b90 msgr2=0x7f58cc0f2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:03.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.659+0000 7f58d098b700 1 -- 172.21.15.148:0/2989965372 shutdown_connections 2023-12-21T03:52:03.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:03.659+0000 7f58d098b700 1 -- 172.21.15.148:0/2989965372 wait complete. 2023-12-21T03:52:03.811 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:01.638308+0000 mgr.smithi078.zidbgq (mgr.14162) 419 : cluster [DBG] pgmap v393: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:01.638308+0000 mgr.smithi078.zidbgq (mgr.14162) 419 : cluster [DBG] pgmap v393: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:03.981 DEBUG:teuthology.orchestra.run.smithi148:osd.4> sudo journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.4.service 2023-12-21T03:52:03.983 INFO:tasks.cephadm:Deploying osd.5 on smithi148 with /dev/vg_nvme/lv_2... 2023-12-21T03:52:03.984 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-21T03:52:04.013 INFO:journalctl@ceph.osd.4.smithi148.stdout:-- Logs begin at Thu 2023-12-21 03:26:40 UTC. -- 2023-12-21T03:52:04.371 INFO:journalctl@ceph.osd.4.smithi148.stdout:Dec 21 03:52:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.4[137176]: debug 2023-12-21T03:52:04.089+0000 7f06724f2080 -1 Falling back to public interface 2023-12-21T03:52:04.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:03.638622+0000 mgr.smithi078.zidbgq (mgr.14162) 420 : cluster [DBG] pgmap v394: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:04.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:03.654255+0000 mon.smithi078 (mon.0) 445 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:52:04.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:03.658139+0000 mon.smithi078 (mon.0) 446 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:52:05.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:03.638622+0000 mgr.smithi078.zidbgq (mgr.14162) 420 : cluster [DBG] pgmap v394: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:05.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:03.654255+0000 mon.smithi078 (mon.0) 445 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:52:05.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:03.658139+0000 mon.smithi078 (mon.0) 446 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:52:05.397 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-21T03:52:05.397 INFO:teuthology.orchestra.run.smithi148.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-21T03:52:05.397 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10+0 records in 2023-12-21T03:52:05.397 INFO:teuthology.orchestra.run.smithi148.stderr:10+0 records out 2023-12-21T03:52:05.398 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121737 s, 861 MB/s 2023-12-21T03:52:05.398 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping successful for: 2023-12-21T03:52:05.806 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi148:vg_nvme/lv_2 2023-12-21T03:52:06.803 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:06 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:05.638877+0000 mgr.smithi078.zidbgq (mgr.14162) 421 : cluster [DBG] pgmap v395: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:06.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.987+0000 7fdfe5730700 1 -- 172.21.15.148:0/564007525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdfe00f6ff0 msgr2=0x7fdfe00f7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.987+0000 7fdfe5730700 1 --2- 172.21.15.148:0/564007525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdfe00f6ff0 0x7fdfe00f7a20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fdfd0004660 tx=0x7fdfd0009b30).stop 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.987+0000 7fdfe5730700 1 -- 172.21.15.148:0/564007525 shutdown_connections 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.987+0000 7fdfe5730700 1 --2- 172.21.15.148:0/564007525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdfe00f6ff0 0x7fdfe00f7a20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.987+0000 7fdfe5730700 1 --2- 172.21.15.148:0/564007525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfe00f66e0 0x7fdfe00f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.987+0000 7fdfe5730700 1 -- 172.21.15.148:0/564007525 >> 172.21.15.148:0/564007525 conn(0x7fdfe00f1f00 msgr2=0x7fdfe00f4320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.987+0000 7fdfe5730700 1 -- 172.21.15.148:0/564007525 shutdown_connections 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.987+0000 7fdfe5730700 1 -- 172.21.15.148:0/564007525 wait complete. 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.987+0000 7fdfe5730700 1 Processor -- start 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfe5730700 1 -- start start 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfe5730700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdfe00f66e0 0x7fdfe006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfe5730700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfe00f6ff0 0x7fdfe0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfe5730700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe006a380 con 0x7fdfe00f6ff0 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfe5730700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe006a4c0 con 0x7fdfe00f66e0 2023-12-21T03:52:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfdeffd700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdfe00f66e0 0x7fdfe006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfdeffd700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdfe00f66e0 0x7fdfe006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35060/0 (socket says 172.21.15.148:35060) 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfdeffd700 1 -- 172.21.15.148:0/149408494 learned_addr learned my addr 172.21.15.148:0/149408494 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfde7fc700 1 --2- 172.21.15.148:0/149408494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfe00f6ff0 0x7fdfe0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfdeffd700 1 -- 172.21.15.148:0/149408494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfe00f6ff0 msgr2=0x7fdfe0069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfdeffd700 1 --2- 172.21.15.148:0/149408494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfe00f6ff0 0x7fdfe0069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfdeffd700 1 -- 172.21.15.148:0/149408494 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd0005040 con 0x7fdfe00f66e0 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfdeffd700 1 --2- 172.21.15.148:0/149408494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdfe00f66e0 0x7fdfe006b9a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fdfd400c790 tx=0x7fdfd4007d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfc77fe700 1 -- 172.21.15.148:0/149408494 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd400e680 con 0x7fdfe00f66e0 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfc77fe700 1 -- 172.21.15.148:0/149408494 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfd400ec90 con 0x7fdfe00f66e0 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.988+0000 7fdfe5730700 1 -- 172.21.15.148:0/149408494 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe006a7a0 con 0x7fdfe00f66e0 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.989+0000 7fdfc77fe700 1 -- 172.21.15.148:0/149408494 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd4013d60 con 0x7fdfe00f66e0 2023-12-21T03:52:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.989+0000 7fdfe5730700 1 -- 172.21.15.148:0/149408494 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe010efd0 con 0x7fdfe00f66e0 2023-12-21T03:52:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.990+0000 7fdfe5730700 1 -- 172.21.15.148:0/149408494 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe0045ad0 con 0x7fdfe00f66e0 2023-12-21T03:52:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.990+0000 7fdfc77fe700 1 -- 172.21.15.148:0/149408494 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdfd401a060 con 0x7fdfe00f66e0 2023-12-21T03:52:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.990+0000 7fdfc77fe700 1 --2- 172.21.15.148:0/149408494 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfc805c2b0 0x7fdfc805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.990+0000 7fdfc77fe700 1 -- 172.21.15.148:0/149408494 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3803+0+0 (secure 0 0 0) 0x7fdfd4075a30 con 0x7fdfe00f66e0 2023-12-21T03:52:06.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.990+0000 7fdfde7fc700 1 --2- 172.21.15.148:0/149408494 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfc805c2b0 0x7fdfc805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:06.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.991+0000 7fdfde7fc700 1 --2- 172.21.15.148:0/149408494 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfc805c2b0 0x7fdfc805e770 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fdfd0013580 tx=0x7fdfd0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:06.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:06.994+0000 7fdfc77fe700 1 -- 172.21.15.148:0/149408494 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfd4048da0 con 0x7fdfe00f66e0 2023-12-21T03:52:07.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:06 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:05.638877+0000 mgr.smithi078.zidbgq (mgr.14162) 421 : cluster [DBG] pgmap v395: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:07.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:07.173+0000 7fdfe5730700 1 -- 172.21.15.148:0/149408494 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfe00fef80 con 0x7fdfc805c2b0 2023-12-21T03:52:07.765 INFO:journalctl@ceph.osd.4.smithi148.stdout:Dec 21 03:52:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.4[137176]: debug 2023-12-21T03:52:07.604+0000 7f06724f2080 -1 osd.4 0 log_to_monitors {default=true} 2023-12-21T03:52:07.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:07.175931+0000 mgr.smithi078.zidbgq (mgr.14162) 422 : audit [DBG] from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:52:07.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:07.176853+0000 mon.smithi078 (mon.0) 447 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:52:07.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:52:07.178328+0000 mgr.smithi078.zidbgq (mgr.14162) 423 : cephadm [INF] Applying service osd.None on host smithi148... 2023-12-21T03:52:07.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:07.178646+0000 mon.smithi078 (mon.0) 448 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:52:07.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:07.179407+0000 mon.smithi078 (mon.0) 449 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:52:07.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:07.609679+0000 mon.smithi148 (mon.1) 10 : audit [INF] from='osd.4 [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-21T03:52:07.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:07.610232+0000 mon.smithi078 (mon.0) 450 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-21T03:52:08.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:07.175931+0000 mgr.smithi078.zidbgq (mgr.14162) 422 : audit [DBG] from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-21T03:52:08.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:07.176853+0000 mon.smithi078 (mon.0) 447 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-21T03:52:08.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:52:07.178328+0000 mgr.smithi078.zidbgq (mgr.14162) 423 : cephadm [INF] Applying service osd.None on host smithi148... 2023-12-21T03:52:08.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:07.178646+0000 mon.smithi078 (mon.0) 448 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-21T03:52:08.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:07.179407+0000 mon.smithi078 (mon.0) 449 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:52:08.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:07.609679+0000 mon.smithi148 (mon.1) 10 : audit [INF] from='osd.4 [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-21T03:52:08.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:07.610232+0000 mon.smithi078 (mon.0) 450 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-21T03:52:09.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:07.639342+0000 mgr.smithi078.zidbgq (mgr.14162) 424 : cluster [DBG] pgmap v396: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:09.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:07.664981+0000 mon.smithi078 (mon.0) 451 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-21T03:52:09.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:07.665037+0000 mon.smithi078 (mon.0) 452 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2023-12-21T03:52:09.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:07.665156+0000 mon.smithi078 (mon.0) 453 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-21T03:52:09.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:07.665513+0000 mon.smithi148 (mon.1) 11 : audit [INF] from='osd.4 [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:52:09.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:07.665826+0000 mon.smithi078 (mon.0) 454 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:52:09.016 INFO:journalctl@ceph.osd.4.smithi148.stdout:Dec 21 03:52:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.4[137176]: debug 2023-12-21T03:52:08.669+0000 7f0668a63700 -1 osd.4 0 waiting for initial osdmap 2023-12-21T03:52:09.016 INFO:journalctl@ceph.osd.4.smithi148.stdout:Dec 21 03:52:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.4[137176]: debug 2023-12-21T03:52:08.677+0000 7f06631db700 -1 osd.4 30 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2023-12-21T03:52:09.061 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:07.639342+0000 mgr.smithi078.zidbgq (mgr.14162) 424 : cluster [DBG] pgmap v396: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:09.061 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:07.664981+0000 mon.smithi078 (mon.0) 451 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-21T03:52:09.061 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:07.665037+0000 mon.smithi078 (mon.0) 452 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2023-12-21T03:52:09.061 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:07.665156+0000 mon.smithi078 (mon.0) 453 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-21T03:52:09.061 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:07.665513+0000 mon.smithi148 (mon.1) 11 : audit [INF] from='osd.4 [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:52:09.062 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:07.665826+0000 mon.smithi078 (mon.0) 454 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:52:10.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:08.666884+0000 mon.smithi078 (mon.0) 455 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2023-12-21T03:52:10.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:08.666999+0000 mon.smithi078 (mon.0) 456 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2023-12-21T03:52:10.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:08.667558+0000 mon.smithi078 (mon.0) 457 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-21T03:52:10.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:08.671414+0000 mon.smithi078 (mon.0) 458 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-21T03:52:10.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:08.760058+0000 mon.smithi148 (mon.1) 12 : audit [INF] from='client.? 172.21.15.148:0/2332685442' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b89bd7a-f664-40fe-8501-4bcca0aa1298"}]: dispatch 2023-12-21T03:52:10.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:08.760612+0000 mon.smithi078 (mon.0) 459 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b89bd7a-f664-40fe-8501-4bcca0aa1298"}]: dispatch 2023-12-21T03:52:10.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:08.765033+0000 mon.smithi078 (mon.0) 460 : cluster [INF] osd.4 [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721] boot 2023-12-21T03:52:10.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:08.765085+0000 mon.smithi078 (mon.0) 461 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b89bd7a-f664-40fe-8501-4bcca0aa1298"}]': finished 2023-12-21T03:52:10.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:08.765162+0000 mon.smithi078 (mon.0) 462 : cluster [DBG] osdmap e31: 6 total, 5 up, 6 in 2023-12-21T03:52:10.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:08.765295+0000 mon.smithi078 (mon.0) 463 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-21T03:52:10.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:08.765768+0000 mon.smithi078 (mon.0) 464 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:10.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:09.188602+0000 mon.smithi148 (mon.1) 13 : audit [DBG] from='client.? 172.21.15.148:0/2740316849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:52:10.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:08.666884+0000 mon.smithi078 (mon.0) 455 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2023-12-21T03:52:10.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:08.666999+0000 mon.smithi078 (mon.0) 456 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2023-12-21T03:52:10.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:08.667558+0000 mon.smithi078 (mon.0) 457 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-21T03:52:10.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:08.671414+0000 mon.smithi078 (mon.0) 458 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-21T03:52:10.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:08.760058+0000 mon.smithi148 (mon.1) 12 : audit [INF] from='client.? 172.21.15.148:0/2332685442' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b89bd7a-f664-40fe-8501-4bcca0aa1298"}]: dispatch 2023-12-21T03:52:10.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:08.760612+0000 mon.smithi078 (mon.0) 459 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b89bd7a-f664-40fe-8501-4bcca0aa1298"}]: dispatch 2023-12-21T03:52:10.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:08.765033+0000 mon.smithi078 (mon.0) 460 : cluster [INF] osd.4 [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721] boot 2023-12-21T03:52:10.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:08.765085+0000 mon.smithi078 (mon.0) 461 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b89bd7a-f664-40fe-8501-4bcca0aa1298"}]': finished 2023-12-21T03:52:10.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:08.765162+0000 mon.smithi078 (mon.0) 462 : cluster [DBG] osdmap e31: 6 total, 5 up, 6 in 2023-12-21T03:52:10.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:08.765295+0000 mon.smithi078 (mon.0) 463 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-21T03:52:10.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:08.765768+0000 mon.smithi078 (mon.0) 464 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:10.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:09.188602+0000 mon.smithi148 (mon.1) 13 : audit [DBG] from='client.? 172.21.15.148:0/2740316849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-21T03:52:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:08.560125+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:52:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:08.560230+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:52:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:09.639761+0000 mgr.smithi078.zidbgq (mgr.14162) 425 : cluster [DBG] pgmap v400: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:10.383407+0000 mon.smithi078 (mon.0) 465 : cluster [DBG] osdmap e32: 6 total, 5 up, 6 in 2023-12-21T03:52:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:10.383543+0000 mon.smithi078 (mon.0) 466 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:08.560125+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:52:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:08.560230+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:52:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:09.639761+0000 mgr.smithi078.zidbgq (mgr.14162) 425 : cluster [DBG] pgmap v400: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-12-21T03:52:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:10.383407+0000 mon.smithi078 (mon.0) 465 : cluster [DBG] osdmap e32: 6 total, 5 up, 6 in 2023-12-21T03:52:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:10.383543+0000 mon.smithi078 (mon.0) 466 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:13.701 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:11.640244+0000 mgr.smithi078.zidbgq (mgr.14162) 426 : cluster [DBG] pgmap v402: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:11.640244+0000 mgr.smithi078.zidbgq (mgr.14162) 426 : cluster [DBG] pgmap v402: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:15.702 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:13.640560+0000 mgr.smithi078.zidbgq (mgr.14162) 427 : cluster [DBG] pgmap v403: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:15.702 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:14.473448+0000 mon.smithi078 (mon.0) 467 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-21T03:52:15.703 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:14.474182+0000 mon.smithi078 (mon.0) 468 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:52:15.703 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:52:14.474762+0000 mgr.smithi078.zidbgq (mgr.14162) 428 : cephadm [INF] Deploying daemon osd.5 on smithi148 2023-12-21T03:52:15.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:13.640560+0000 mgr.smithi078.zidbgq (mgr.14162) 427 : cluster [DBG] pgmap v403: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:15.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:14.473448+0000 mon.smithi078 (mon.0) 467 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-21T03:52:15.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:14.474182+0000 mon.smithi078 (mon.0) 468 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:52:15.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:52:14.474762+0000 mgr.smithi078.zidbgq (mgr.14162) 428 : cephadm [INF] Deploying daemon osd.5 on smithi148 2023-12-21T03:52:16.685 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 5 on host 'smithi148' 2023-12-21T03:52:16.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.683+0000 7fdfc77fe700 1 -- 172.21.15.148:0/149408494 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdfe00fef80 con 0x7fdfc805c2b0 2023-12-21T03:52:16.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.685+0000 7fdfe5730700 1 -- 172.21.15.148:0/149408494 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfc805c2b0 msgr2=0x7fdfc805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:16.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.685+0000 7fdfe5730700 1 --2- 172.21.15.148:0/149408494 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfc805c2b0 0x7fdfc805e770 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fdfd0013580 tx=0x7fdfd0009b30).stop 2023-12-21T03:52:16.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.685+0000 7fdfe5730700 1 -- 172.21.15.148:0/149408494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdfe00f66e0 msgr2=0x7fdfe006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:16.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.685+0000 7fdfe5730700 1 --2- 172.21.15.148:0/149408494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdfe00f66e0 0x7fdfe006b9a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fdfd400c790 tx=0x7fdfd4007d70).stop 2023-12-21T03:52:16.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.685+0000 7fdfe5730700 1 -- 172.21.15.148:0/149408494 shutdown_connections 2023-12-21T03:52:16.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.685+0000 7fdfe5730700 1 --2- 172.21.15.148:0/149408494 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fdfc805c2b0 0x7fdfc805e770 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:16.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.685+0000 7fdfe5730700 1 --2- 172.21.15.148:0/149408494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdfe00f66e0 0x7fdfe006b9a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:16.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.685+0000 7fdfe5730700 1 --2- 172.21.15.148:0/149408494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfe00f6ff0 0x7fdfe0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:16.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.685+0000 7fdfe5730700 1 -- 172.21.15.148:0/149408494 >> 172.21.15.148:0/149408494 conn(0x7fdfe00f1f00 msgr2=0x7fdfe00fb470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:16.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.685+0000 7fdfe5730700 1 -- 172.21.15.148:0/149408494 shutdown_connections 2023-12-21T03:52:16.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:16.685+0000 7fdfe5730700 1 -- 172.21.15.148:0/149408494 wait complete. 2023-12-21T03:52:17.001 DEBUG:teuthology.orchestra.run.smithi148:osd.5> sudo journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.5.service 2023-12-21T03:52:17.004 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2023-12-21T03:52:17.004 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-21T03:52:17.034 INFO:journalctl@ceph.osd.5.smithi148.stdout:-- Logs begin at Thu 2023-12-21 03:26:40 UTC. -- 2023-12-21T03:52:17.265 INFO:journalctl@ceph.osd.5.smithi148.stdout:Dec 21 03:52:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.5[140324]: debug 2023-12-21T03:52:17.136+0000 7fa61a288080 -1 Falling back to public interface 2023-12-21T03:52:17.653 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:15.640945+0000 mgr.smithi078.zidbgq (mgr.14162) 429 : cluster [DBG] pgmap v404: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:17.654 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:16.677041+0000 mon.smithi078 (mon.0) 469 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:52:17.654 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:16.683197+0000 mon.smithi078 (mon.0) 470 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:52:17.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:15.640945+0000 mgr.smithi078.zidbgq (mgr.14162) 429 : cluster [DBG] pgmap v404: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:17.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:16.677041+0000 mon.smithi078 (mon.0) 469 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:52:17.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:16.683197+0000 mon.smithi078 (mon.0) 470 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:52:18.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.181+0000 7f2322260700 1 -- 172.21.15.78:0/3553589876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c0f68e0 msgr2=0x7f231c0f6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.181+0000 7f2322260700 1 --2- 172.21.15.78:0/3553589876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c0f68e0 0x7f231c0f6d00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f230c004660 tx=0x7f230c009b30).stop 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.181+0000 7f2322260700 1 -- 172.21.15.78:0/3553589876 shutdown_connections 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.181+0000 7f2322260700 1 --2- 172.21.15.78:0/3553589876 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0f7b80 0x7f231c0f7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.181+0000 7f2322260700 1 --2- 172.21.15.78:0/3553589876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c0f68e0 0x7f231c0f6d00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.181+0000 7f2322260700 1 -- 172.21.15.78:0/3553589876 >> 172.21.15.78:0/3553589876 conn(0x7f231c0f2020 msgr2=0x7f231c0f4460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.182+0000 7f2322260700 1 -- 172.21.15.78:0/3553589876 shutdown_connections 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.182+0000 7f2322260700 1 -- 172.21.15.78:0/3553589876 wait complete. 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.182+0000 7f2322260700 1 Processor -- start 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.182+0000 7f2322260700 1 -- start start 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.182+0000 7f2322260700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0f68e0 0x7f231c1071a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.182+0000 7f2322260700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c0f7b80 0x7f231c107690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.182+0000 7f2322260700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c107cd0 con 0x7f231c0f7b80 2023-12-21T03:52:18.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.182+0000 7f2322260700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c108580 con 0x7f231c0f68e0 2023-12-21T03:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.182+0000 7f231affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c0f7b80 0x7f231c107690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f231affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c0f7b80 0x7f231c107690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39012/0 (socket says 172.21.15.78:39012) 2023-12-21T03:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f231affd700 1 -- 172.21.15.78:0/4160045639 learned_addr learned my addr 172.21.15.78:0/4160045639 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f231b7fe700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0f68e0 0x7f231c1071a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f231affd700 1 -- 172.21.15.78:0/4160045639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0f68e0 msgr2=0x7f231c1071a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f231affd700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0f68e0 0x7f231c1071a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f231affd700 1 -- 172.21.15.78:0/4160045639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c005040 con 0x7f231c0f7b80 2023-12-21T03:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f231b7fe700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0f68e0 0x7f231c1071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f231affd700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c0f7b80 0x7f231c107690 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f231000b410 tx=0x7f2310006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f2303fff700 1 -- 172.21.15.78:0/4160045639 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23100125f0 con 0x7f231c0f7b80 2023-12-21T03:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f2322260700 1 -- 172.21.15.78:0/4160045639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231c108860 con 0x7f231c0f7b80 2023-12-21T03:52:18.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f2303fff700 1 -- 172.21.15.78:0/4160045639 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2310012c00 con 0x7f231c0f7b80 2023-12-21T03:52:18.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f2303fff700 1 -- 172.21.15.78:0/4160045639 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2310002aa0 con 0x7f231c0f7b80 2023-12-21T03:52:18.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.183+0000 7f2322260700 1 -- 172.21.15.78:0/4160045639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231c108e20 con 0x7f231c0f7b80 2023-12-21T03:52:18.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.184+0000 7f2322260700 1 -- 172.21.15.78:0/4160045639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f231c0f5cd0 con 0x7f231c0f7b80 2023-12-21T03:52:18.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.184+0000 7f2303fff700 1 -- 172.21.15.78:0/4160045639 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f231000f030 con 0x7f231c0f7b80 2023-12-21T03:52:18.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.185+0000 7f2303fff700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f230405c2b0 0x7f230405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:18.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.185+0000 7f2303fff700 1 -- 172.21.15.78:0/4160045639 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4235+0+0 (secure 0 0 0) 0x7f2310075f20 con 0x7f231c0f7b80 2023-12-21T03:52:18.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.185+0000 7f231b7fe700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f230405c2b0 0x7f230405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:18.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.185+0000 7f231b7fe700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f230405c2b0 0x7f230405e770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f230c009d60 tx=0x7f230c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:18.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.189+0000 7f2303fff700 1 -- 172.21.15.78:0/4160045639 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2310049000 con 0x7f231c0f7b80 2023-12-21T03:52:18.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.340+0000 7f2322260700 1 -- 172.21.15.78:0/4160045639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f231c027aa0 con 0x7f231c0f7b80 2023-12-21T03:52:18.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.340+0000 7f2303fff700 1 -- 172.21.15.78:0/4160045639 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v32) v1 ==== 74+0+130 (secure 0 0 0) 0x7f231004c620 con 0x7f231c0f7b80 2023-12-21T03:52:18.342 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:52:18.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.342+0000 7f2322260700 1 -- 172.21.15.78:0/4160045639 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f230405c2b0 msgr2=0x7f230405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:18.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.342+0000 7f2322260700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f230405c2b0 0x7f230405e770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f230c009d60 tx=0x7f230c009b30).stop 2023-12-21T03:52:18.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.342+0000 7f2322260700 1 -- 172.21.15.78:0/4160045639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c0f7b80 msgr2=0x7f231c107690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:18.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.342+0000 7f2322260700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c0f7b80 0x7f231c107690 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f231000b410 tx=0x7f2310006ef0).stop 2023-12-21T03:52:18.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.342+0000 7f2322260700 1 -- 172.21.15.78:0/4160045639 shutdown_connections 2023-12-21T03:52:18.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.342+0000 7f2322260700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f230405c2b0 0x7f230405e770 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:18.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.342+0000 7f2322260700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0f68e0 0x7f231c1071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:18.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.342+0000 7f2322260700 1 --2- 172.21.15.78:0/4160045639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c0f7b80 0x7f231c107690 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:18.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.342+0000 7f2322260700 1 -- 172.21.15.78:0/4160045639 >> 172.21.15.78:0/4160045639 conn(0x7f231c0f2020 msgr2=0x7f231c0fb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:18.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.342+0000 7f2322260700 1 -- 172.21.15.78:0/4160045639 shutdown_connections 2023-12-21T03:52:18.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:18.342+0000 7f2322260700 1 -- 172.21.15.78:0/4160045639 wait complete. 2023-12-21T03:52:18.664 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:18 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:18.341953+0000 mon.smithi078 (mon.0) 471 : audit [DBG] from='client.? 172.21.15.78:0/4160045639' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-21T03:52:18.731 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"num_osds":6,"num_up_osds":5,"osd_up_since":1703130728,"num_in_osds":6,"osd_in_since":1703130728,"num_remapped_pgs":0} 2023-12-21T03:52:18.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:18 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:18.341953+0000 mon.smithi078 (mon.0) 471 : audit [DBG] from='client.? 172.21.15.78:0/4160045639' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-21T03:52:19.733 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-21T03:52:19.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:17.641309+0000 mgr.smithi078.zidbgq (mgr.14162) 430 : cluster [DBG] pgmap v405: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:19.767 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:17.641309+0000 mgr.smithi078.zidbgq (mgr.14162) 430 : cluster [DBG] pgmap v405: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.984+0000 7f54e5c55700 1 -- 172.21.15.78:0/717060509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54e00f5430 msgr2=0x7f54e00f5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.984+0000 7f54e5c55700 1 --2- 172.21.15.78:0/717060509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54e00f5430 0x7f54e00f5850 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f54d4004660 tx=0x7f54d4009b30).stop 2023-12-21T03:52:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.984+0000 7f54e5c55700 1 -- 172.21.15.78:0/717060509 shutdown_connections 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.984+0000 7f54e5c55700 1 --2- 172.21.15.78:0/717060509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e00f66d0 0x7f54e00f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.984+0000 7f54e5c55700 1 --2- 172.21.15.78:0/717060509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54e00f5430 0x7f54e00f5850 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.984+0000 7f54e5c55700 1 -- 172.21.15.78:0/717060509 >> 172.21.15.78:0/717060509 conn(0x7f54e00f0b90 msgr2=0x7f54e00f2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.984+0000 7f54e5c55700 1 -- 172.21.15.78:0/717060509 shutdown_connections 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.984+0000 7f54e5c55700 1 -- 172.21.15.78:0/717060509 wait complete. 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e5c55700 1 Processor -- start 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e5c55700 1 -- start start 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e5c55700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54e00f5430 0x7f54e0109af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e5c55700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e00f66d0 0x7f54e010bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e5c55700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e0060450 con 0x7f54e00f5430 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e5c55700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e0060590 con 0x7f54e00f66d0 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e4c53700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54e00f5430 0x7f54e0109af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e4c53700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54e00f5430 0x7f54e0109af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53366/0 (socket says 172.21.15.78:53366) 2023-12-21T03:52:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e4c53700 1 -- 172.21.15.78:0/243546849 learned_addr learned my addr 172.21.15.78:0/243546849 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54dffff700 1 --2- 172.21.15.78:0/243546849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e00f66d0 0x7f54e010bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e4c53700 1 -- 172.21.15.78:0/243546849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e00f66d0 msgr2=0x7f54e010bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e4c53700 1 --2- 172.21.15.78:0/243546849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e00f66d0 0x7f54e010bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e4c53700 1 -- 172.21.15.78:0/243546849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d4005040 con 0x7f54e00f5430 2023-12-21T03:52:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.985+0000 7f54e4c53700 1 --2- 172.21.15.78:0/243546849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54e00f5430 0x7f54e0109af0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f54d4009f50 tx=0x7f54d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.986+0000 7f54dd7fa700 1 -- 172.21.15.78:0/243546849 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54d4018070 con 0x7f54e00f5430 2023-12-21T03:52:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.986+0000 7f54dd7fa700 1 -- 172.21.15.78:0/243546849 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54d4013410 con 0x7f54e00f5430 2023-12-21T03:52:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.986+0000 7f54e5c55700 1 -- 172.21.15.78:0/243546849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e010c5a0 con 0x7f54e00f5430 2023-12-21T03:52:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.986+0000 7f54dd7fa700 1 -- 172.21.15.78:0/243546849 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54d401c620 con 0x7f54e00f5430 2023-12-21T03:52:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.986+0000 7f54e5c55700 1 -- 172.21.15.78:0/243546849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e010cb00 con 0x7f54e00f5430 2023-12-21T03:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.987+0000 7f54e5c55700 1 -- 172.21.15.78:0/243546849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e0045ad0 con 0x7f54e00f5430 2023-12-21T03:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.987+0000 7f54dd7fa700 1 -- 172.21.15.78:0/243546849 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f54d401d420 con 0x7f54e00f5430 2023-12-21T03:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.987+0000 7f54dd7fa700 1 --2- 172.21.15.78:0/243546849 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f54d005c300 0x7f54d005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.987+0000 7f54dd7fa700 1 -- 172.21.15.78:0/243546849 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4235+0+0 (secure 0 0 0) 0x7f54d4077970 con 0x7f54e00f5430 2023-12-21T03:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.987+0000 7f54dffff700 1 --2- 172.21.15.78:0/243546849 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f54d005c300 0x7f54d005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.988+0000 7f54dffff700 1 --2- 172.21.15.78:0/243546849 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f54d005c300 0x7f54d005e7c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f54cc00b790 tx=0x7f54cc008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:20.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:20.991+0000 7f54dd7fa700 1 -- 172.21.15.78:0/243546849 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54d404aa50 con 0x7f54e00f5430 2023-12-21T03:52:21.015 INFO:journalctl@ceph.osd.5.smithi148.stdout:Dec 21 03:52:20 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.5[140324]: debug 2023-12-21T03:52:20.580+0000 7fa61a288080 -1 osd.5 0 log_to_monitors {default=true} 2023-12-21T03:52:21.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.142+0000 7f54e5c55700 1 -- 172.21.15.78:0/243546849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f54e0027a00 con 0x7f54e00f5430 2023-12-21T03:52:21.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.142+0000 7f54dd7fa700 1 -- 172.21.15.78:0/243546849 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v32) v1 ==== 74+0+130 (secure 0 0 0) 0x7f54d404e070 con 0x7f54e00f5430 2023-12-21T03:52:21.143 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:52:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.143+0000 7f54e5c55700 1 -- 172.21.15.78:0/243546849 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f54d005c300 msgr2=0x7f54d005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.143+0000 7f54e5c55700 1 --2- 172.21.15.78:0/243546849 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f54d005c300 0x7f54d005e7c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f54cc00b790 tx=0x7f54cc008040).stop 2023-12-21T03:52:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.143+0000 7f54e5c55700 1 -- 172.21.15.78:0/243546849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54e00f5430 msgr2=0x7f54e0109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.143+0000 7f54e5c55700 1 --2- 172.21.15.78:0/243546849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54e00f5430 0x7f54e0109af0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f54d4009f50 tx=0x7f54d4009b30).stop 2023-12-21T03:52:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.143+0000 7f54e5c55700 1 -- 172.21.15.78:0/243546849 shutdown_connections 2023-12-21T03:52:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.143+0000 7f54e5c55700 1 --2- 172.21.15.78:0/243546849 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f54d005c300 0x7f54d005e7c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.143+0000 7f54e5c55700 1 --2- 172.21.15.78:0/243546849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54e00f5430 0x7f54e0109af0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.143+0000 7f54e5c55700 1 --2- 172.21.15.78:0/243546849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e00f66d0 0x7f54e010bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.144+0000 7f54e5c55700 1 -- 172.21.15.78:0/243546849 >> 172.21.15.78:0/243546849 conn(0x7f54e00f0b90 msgr2=0x7f54e00fa120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.144+0000 7f54e5c55700 1 -- 172.21.15.78:0/243546849 shutdown_connections 2023-12-21T03:52:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:21.144+0000 7f54e5c55700 1 -- 172.21.15.78:0/243546849 wait complete. 2023-12-21T03:52:21.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:19.641670+0000 mgr.smithi078.zidbgq (mgr.14162) 431 : cluster [DBG] pgmap v406: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:21.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:20.584564+0000 mon.smithi148 (mon.1) 14 : audit [INF] from='osd.5 [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-21T03:52:21.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:20.584925+0000 mon.smithi078 (mon.0) 472 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-21T03:52:21.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:21.143359+0000 mon.smithi078 (mon.0) 473 : audit [DBG] from='client.? 172.21.15.78:0/243546849' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-21T03:52:21.505 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"num_osds":6,"num_up_osds":5,"osd_up_since":1703130728,"num_in_osds":6,"osd_in_since":1703130728,"num_remapped_pgs":0} 2023-12-21T03:52:21.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:19.641670+0000 mgr.smithi078.zidbgq (mgr.14162) 431 : cluster [DBG] pgmap v406: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:21.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:20.584564+0000 mon.smithi148 (mon.1) 14 : audit [INF] from='osd.5 [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-21T03:52:21.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:20.584925+0000 mon.smithi078 (mon.0) 472 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-21T03:52:21.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:21.143359+0000 mon.smithi078 (mon.0) 473 : audit [DBG] from='client.? 172.21.15.78:0/243546849' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-21T03:52:22.506 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-21T03:52:22.766 INFO:journalctl@ceph.osd.5.smithi148.stdout:Dec 21 03:52:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.5[140324]: debug 2023-12-21T03:52:22.415+0000 7fa6107f9700 -1 osd.5 0 waiting for initial osdmap 2023-12-21T03:52:22.766 INFO:journalctl@ceph.osd.5.smithi148.stdout:Dec 21 03:52:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.5[140324]: debug 2023-12-21T03:52:22.424+0000 7fa608f6d700 -1 osd.5 34 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2023-12-21T03:52:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:21.404506+0000 mon.smithi078 (mon.0) 474 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-21T03:52:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:21.404579+0000 mon.smithi078 (mon.0) 475 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2023-12-21T03:52:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:21.404955+0000 mon.smithi078 (mon.0) 476 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:21.406403+0000 mon.smithi148 (mon.1) 15 : audit [INF] from='osd.5 [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:52:22.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:22 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:21.409464+0000 mon.smithi078 (mon.0) 477 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:52:22.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:21.404506+0000 mon.smithi078 (mon.0) 474 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-21T03:52:22.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:21.404579+0000 mon.smithi078 (mon.0) 475 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2023-12-21T03:52:22.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:21.404955+0000 mon.smithi078 (mon.0) 476 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:22.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:21.406403+0000 mon.smithi148 (mon.1) 15 : audit [INF] from='osd.5 [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:52:22.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:22 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:21.409464+0000 mon.smithi078 (mon.0) 477 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2023-12-21T03:52:23.536 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:21.642145+0000 mgr.smithi078.zidbgq (mgr.14162) 432 : cluster [DBG] pgmap v408: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:23.536 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:22.411496+0000 mon.smithi078 (mon.0) 478 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2023-12-21T03:52:23.536 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:22.411602+0000 mon.smithi078 (mon.0) 479 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2023-12-21T03:52:23.536 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:22.412111+0000 mon.smithi078 (mon.0) 480 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:23.536 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:22.415966+0000 mon.smithi078 (mon.0) 481 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:23.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.717+0000 7f825e17b700 1 -- 172.21.15.78:0/1779337358 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82580f55c0 msgr2=0x7f82580f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:23.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.717+0000 7f825e17b700 1 --2- 172.21.15.78:0/1779337358 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82580f55c0 0x7f82580f59e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f8248004660 tx=0x7f8248009b30).stop 2023-12-21T03:52:23.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.717+0000 7f825e17b700 1 -- 172.21.15.78:0/1779337358 shutdown_connections 2023-12-21T03:52:23.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.717+0000 7f825e17b700 1 --2- 172.21.15.78:0/1779337358 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82580f6860 0x7f82580f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:23.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.717+0000 7f825e17b700 1 --2- 172.21.15.78:0/1779337358 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82580f55c0 0x7f82580f59e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:23.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.717+0000 7f825e17b700 1 -- 172.21.15.78:0/1779337358 >> 172.21.15.78:0/1779337358 conn(0x7f82580f0d40 msgr2=0x7f82580f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:23.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.717+0000 7f825e17b700 1 -- 172.21.15.78:0/1779337358 shutdown_connections 2023-12-21T03:52:23.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.717+0000 7f825e17b700 1 -- 172.21.15.78:0/1779337358 wait complete. 2023-12-21T03:52:23.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f825e17b700 1 Processor -- start 2023-12-21T03:52:23.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f825e17b700 1 -- start start 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f825e17b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82580f55c0 0x7f82580f9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f825e17b700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82580f6860 0x7f82580f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f825e17b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82580f8100 con 0x7f82580f55c0 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f825e17b700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82580f8240 con 0x7f82580f6860 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f82577fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82580f55c0 0x7f82580f9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f82577fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82580f55c0 0x7f82580f9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53394/0 (socket says 172.21.15.78:53394) 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f82577fe700 1 -- 172.21.15.78:0/2473079218 learned_addr learned my addr 172.21.15.78:0/2473079218 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f8256ffd700 1 --2- 172.21.15.78:0/2473079218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82580f6860 0x7f82580f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f82577fe700 1 -- 172.21.15.78:0/2473079218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82580f6860 msgr2=0x7f82580f7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f82577fe700 1 --2- 172.21.15.78:0/2473079218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82580f6860 0x7f82580f7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f82577fe700 1 -- 172.21.15.78:0/2473079218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8248005040 con 0x7f82580f55c0 2023-12-21T03:52:23.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.718+0000 7f82577fe700 1 --2- 172.21.15.78:0/2473079218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82580f55c0 0x7f82580f9720 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f8248009f50 tx=0x7f8248009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:23.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.719+0000 7f824ffff700 1 -- 172.21.15.78:0/2473079218 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8248018070 con 0x7f82580f55c0 2023-12-21T03:52:23.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.719+0000 7f824ffff700 1 -- 172.21.15.78:0/2473079218 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8248013410 con 0x7f82580f55c0 2023-12-21T03:52:23.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.719+0000 7f825e17b700 1 -- 172.21.15.78:0/2473079218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82580f84c0 con 0x7f82580f55c0 2023-12-21T03:52:23.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.719+0000 7f824ffff700 1 -- 172.21.15.78:0/2473079218 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f824801c620 con 0x7f82580f55c0 2023-12-21T03:52:23.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.719+0000 7f825e17b700 1 -- 172.21.15.78:0/2473079218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82580f8a20 con 0x7f82580f55c0 2023-12-21T03:52:23.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.720+0000 7f825e17b700 1 -- 172.21.15.78:0/2473079218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8258045ad0 con 0x7f82580f55c0 2023-12-21T03:52:23.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.720+0000 7f824ffff700 1 -- 172.21.15.78:0/2473079218 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f824801d420 con 0x7f82580f55c0 2023-12-21T03:52:23.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.720+0000 7f824ffff700 1 --2- 172.21.15.78:0/2473079218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f824405c2b0 0x7f824405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:23.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.720+0000 7f824ffff700 1 -- 172.21.15.78:0/2473079218 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8248077a40 con 0x7f82580f55c0 2023-12-21T03:52:23.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.720+0000 7f8256ffd700 1 --2- 172.21.15.78:0/2473079218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f824405c2b0 0x7f824405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:23.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.721+0000 7f8256ffd700 1 --2- 172.21.15.78:0/2473079218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f824405c2b0 0x7f824405e770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f82400041d0 tx=0x7f8240009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:23.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.724+0000 7f824ffff700 1 -- 172.21.15.78:0/2473079218 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f824804aa40 con 0x7f82580f55c0 2023-12-21T03:52:23.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:21.642145+0000 mgr.smithi078.zidbgq (mgr.14162) 432 : cluster [DBG] pgmap v408: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:23.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:22.411496+0000 mon.smithi078 (mon.0) 478 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2023-12-21T03:52:23.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:22.411602+0000 mon.smithi078 (mon.0) 479 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2023-12-21T03:52:23.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:22.412111+0000 mon.smithi078 (mon.0) 480 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:23.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:22.415966+0000 mon.smithi078 (mon.0) 481 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:23.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.876+0000 7f825e17b700 1 -- 172.21.15.78:0/2473079218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f825810f680 con 0x7f82580f55c0 2023-12-21T03:52:23.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.876+0000 7f824ffff700 1 -- 172.21.15.78:0/2473079218 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v35) v1 ==== 74+0+130 (secure 0 0 0) 0x7f824804e060 con 0x7f82580f55c0 2023-12-21T03:52:23.877 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:52:23.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.878+0000 7f825e17b700 1 -- 172.21.15.78:0/2473079218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f824405c2b0 msgr2=0x7f824405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:23.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.878+0000 7f825e17b700 1 --2- 172.21.15.78:0/2473079218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f824405c2b0 0x7f824405e770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f82400041d0 tx=0x7f8240009040).stop 2023-12-21T03:52:23.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.878+0000 7f825e17b700 1 -- 172.21.15.78:0/2473079218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82580f55c0 msgr2=0x7f82580f9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:23.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.878+0000 7f825e17b700 1 --2- 172.21.15.78:0/2473079218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82580f55c0 0x7f82580f9720 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f8248009f50 tx=0x7f8248009b30).stop 2023-12-21T03:52:23.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.878+0000 7f825e17b700 1 -- 172.21.15.78:0/2473079218 shutdown_connections 2023-12-21T03:52:23.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.878+0000 7f825e17b700 1 --2- 172.21.15.78:0/2473079218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f824405c2b0 0x7f824405e770 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:23.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.878+0000 7f825e17b700 1 --2- 172.21.15.78:0/2473079218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82580f55c0 0x7f82580f9720 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:23.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.878+0000 7f825e17b700 1 --2- 172.21.15.78:0/2473079218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82580f6860 0x7f82580f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:23.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.878+0000 7f825e17b700 1 -- 172.21.15.78:0/2473079218 >> 172.21.15.78:0/2473079218 conn(0x7f82580f0d40 msgr2=0x7f825805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:23.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.878+0000 7f825e17b700 1 -- 172.21.15.78:0/2473079218 shutdown_connections 2023-12-21T03:52:23.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:23.878+0000 7f825e17b700 1 -- 172.21.15.78:0/2473079218 wait complete. 2023-12-21T03:52:24.239 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":35,"num_osds":6,"num_up_osds":6,"osd_up_since":1703130743,"num_in_osds":6,"osd_in_since":1703130728,"num_remapped_pgs":0} 2023-12-21T03:52:24.239 INFO:tasks.cephadm:Setting up client nodes... 2023-12-21T03:52:24.240 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-21T03:52:24.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:24 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:21.548859+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:52:24.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:24 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:21.548976+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:52:24.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:24 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:23.417421+0000 mon.smithi078 (mon.0) 482 : cluster [INF] osd.5 [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132] boot 2023-12-21T03:52:24.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:24 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:23.417471+0000 mon.smithi078 (mon.0) 483 : cluster [DBG] osdmap e35: 6 total, 6 up, 6 in 2023-12-21T03:52:24.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:24 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:23.417941+0000 mon.smithi078 (mon.0) 484 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:24.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:24 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:23.877536+0000 mon.smithi078 (mon.0) 485 : audit [DBG] from='client.? 172.21.15.78:0/2473079218' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-21T03:52:24.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:24 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:21.548859+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-21T03:52:24.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:24 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:21.548976+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-21T03:52:24.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:24 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:23.417421+0000 mon.smithi078 (mon.0) 482 : cluster [INF] osd.5 [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132] boot 2023-12-21T03:52:24.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:24 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:23.417471+0000 mon.smithi078 (mon.0) 483 : cluster [DBG] osdmap e35: 6 total, 6 up, 6 in 2023-12-21T03:52:24.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:24 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:23.417941+0000 mon.smithi078 (mon.0) 484 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-21T03:52:24.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:24 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:23.877536+0000 mon.smithi078 (mon.0) 485 : audit [DBG] from='client.? 172.21.15.78:0/2473079218' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-21T03:52:25.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.438+0000 7fd36263b700 1 -- 172.21.15.78:0/3835200297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0f79c0 msgr2=0x7fd35c0f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:25.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.438+0000 7fd36263b700 1 --2- 172.21.15.78:0/3835200297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0f79c0 0x7fd35c0f7e20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fd34c004660 tx=0x7fd34c009b30).stop 2023-12-21T03:52:25.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.438+0000 7fd36263b700 1 -- 172.21.15.78:0/3835200297 shutdown_connections 2023-12-21T03:52:25.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.438+0000 7fd36263b700 1 --2- 172.21.15.78:0/3835200297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0f79c0 0x7fd35c0f7e20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:25.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.438+0000 7fd36263b700 1 --2- 172.21.15.78:0/3835200297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd35c0f6720 0x7fd35c0f6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:25.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.438+0000 7fd36263b700 1 -- 172.21.15.78:0/3835200297 >> 172.21.15.78:0/3835200297 conn(0x7fd35c0f1e80 msgr2=0x7fd35c0f42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:25.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.438+0000 7fd36263b700 1 -- 172.21.15.78:0/3835200297 shutdown_connections 2023-12-21T03:52:25.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.438+0000 7fd36263b700 1 -- 172.21.15.78:0/3835200297 wait complete. 2023-12-21T03:52:25.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.438+0000 7fd36263b700 1 Processor -- start 2023-12-21T03:52:25.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.438+0000 7fd36263b700 1 -- start start 2023-12-21T03:52:25.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd36263b700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd35c0f6720 0x7fd35c109150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:25.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd36263b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0f79c0 0x7fd35c109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:25.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd36263b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c109bf0 con 0x7fd35c0f79c0 2023-12-21T03:52:25.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd36263b700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c109d30 con 0x7fd35c0f6720 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd35b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0f79c0 0x7fd35c109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd35b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0f79c0 0x7fd35c109640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53402/0 (socket says 172.21.15.78:53402) 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd35b7fe700 1 -- 172.21.15.78:0/811143943 learned_addr learned my addr 172.21.15.78:0/811143943 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd35b7fe700 1 -- 172.21.15.78:0/811143943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd35c0f6720 msgr2=0x7fd35c109150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd35b7fe700 1 --2- 172.21.15.78:0/811143943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd35c0f6720 0x7fd35c109150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd35b7fe700 1 -- 172.21.15.78:0/811143943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c005040 con 0x7fd35c0f79c0 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd35b7fe700 1 --2- 172.21.15.78:0/811143943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0f79c0 0x7fd35c109640 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fd34c009f50 tx=0x7fd34c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd358ff9700 1 -- 172.21.15.78:0/811143943 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd34c018070 con 0x7fd35c0f79c0 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd36263b700 1 -- 172.21.15.78:0/811143943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c10a670 con 0x7fd35c0f79c0 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd358ff9700 1 -- 172.21.15.78:0/811143943 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd34c013410 con 0x7fd35c0f79c0 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd358ff9700 1 -- 172.21.15.78:0/811143943 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd34c01c3c0 con 0x7fd35c0f79c0 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.439+0000 7fd36263b700 1 -- 172.21.15.78:0/811143943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c10abd0 con 0x7fd35c0f79c0 2023-12-21T03:52:25.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.440+0000 7fd36263b700 1 -- 172.21.15.78:0/811143943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c045ad0 con 0x7fd35c0f79c0 2023-12-21T03:52:25.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.440+0000 7fd358ff9700 1 -- 172.21.15.78:0/811143943 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd34c01d420 con 0x7fd35c0f79c0 2023-12-21T03:52:25.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.441+0000 7fd358ff9700 1 --2- 172.21.15.78:0/811143943 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd34405c2b0 0x7fd34405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:25.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.441+0000 7fd358ff9700 1 -- 172.21.15.78:0/811143943 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd34c077870 con 0x7fd35c0f79c0 2023-12-21T03:52:25.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.441+0000 7fd35bfff700 1 --2- 172.21.15.78:0/811143943 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd34405c2b0 0x7fd34405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:25.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.441+0000 7fd35bfff700 1 --2- 172.21.15.78:0/811143943 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd34405c2b0 0x7fd34405e770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fd35c0f7820 tx=0x7fd3500099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:25.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.445+0000 7fd358ff9700 1 -- 172.21.15.78:0/811143943 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd34c04a870 con 0x7fd35c0f79c0 2023-12-21T03:52:25.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:23.642396+0000 mgr.smithi078.zidbgq (mgr.14162) 433 : cluster [DBG] pgmap v411: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:25.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:24.425897+0000 mon.smithi078 (mon.0) 486 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2023-12-21T03:52:25.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.652+0000 7fd36263b700 1 -- 172.21.15.78:0/811143943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fd35c027a00 con 0x7fd35c0f79c0 2023-12-21T03:52:25.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.656+0000 7fd358ff9700 1 -- 172.21.15.78:0/811143943 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7fd34c04de90 con 0x7fd35c0f79c0 2023-12-21T03:52:25.658 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2023-12-21T03:52:25.658 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQB5toNl5hb7JhAA88vvzmOVdsnCp75Mb2mfZA== 2023-12-21T03:52:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.658+0000 7fd36263b700 1 -- 172.21.15.78:0/811143943 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd34405c2b0 msgr2=0x7fd34405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.658+0000 7fd36263b700 1 --2- 172.21.15.78:0/811143943 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd34405c2b0 0x7fd34405e770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fd35c0f7820 tx=0x7fd3500099e0).stop 2023-12-21T03:52:25.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.658+0000 7fd36263b700 1 -- 172.21.15.78:0/811143943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0f79c0 msgr2=0x7fd35c109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:25.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.658+0000 7fd36263b700 1 --2- 172.21.15.78:0/811143943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0f79c0 0x7fd35c109640 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fd34c009f50 tx=0x7fd34c009b30).stop 2023-12-21T03:52:25.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.658+0000 7fd36263b700 1 -- 172.21.15.78:0/811143943 shutdown_connections 2023-12-21T03:52:25.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.658+0000 7fd36263b700 1 --2- 172.21.15.78:0/811143943 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd34405c2b0 0x7fd34405e770 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:25.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.658+0000 7fd36263b700 1 --2- 172.21.15.78:0/811143943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd35c0f6720 0x7fd35c109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:52:25.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.658+0000 7fd36263b700 1 --2- 172.21.15.78:0/811143943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0f79c0 0x7fd35c109640 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:25.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.658+0000 7fd36263b700 1 -- 172.21.15.78:0/811143943 >> 172.21.15.78:0/811143943 conn(0x7fd35c0f1e80 msgr2=0x7fd35c0f9400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:25.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.658+0000 7fd36263b700 1 -- 172.21.15.78:0/811143943 shutdown_connections 2023-12-21T03:52:25.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:25.658+0000 7fd36263b700 1 -- 172.21.15.78:0/811143943 wait complete. 2023-12-21T03:52:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:23.642396+0000 mgr.smithi078.zidbgq (mgr.14162) 433 : cluster [DBG] pgmap v411: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-12-21T03:52:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:24.425897+0000 mon.smithi078 (mon.0) 486 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2023-12-21T03:52:26.031 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-12-21T03:52:26.031 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-12-21T03:52:26.031 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-21T03:52:26.071 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-21T03:52:26.733 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:25.653819+0000 mon.smithi078 (mon.0) 487 : audit [INF] from='client.? 172.21.15.78:0/811143943' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-21T03:52:26.733 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:25.657509+0000 mon.smithi078 (mon.0) 488 : audit [INF] from='client.? 172.21.15.78:0/811143943' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-21T03:52:26.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:25.653819+0000 mon.smithi078 (mon.0) 487 : audit [INF] from='client.? 172.21.15.78:0/811143943' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-21T03:52:26.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:25.657509+0000 mon.smithi078 (mon.0) 488 : audit [INF] from='client.? 172.21.15.78:0/811143943' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-21T03:52:27.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.223+0000 7fe968354700 1 -- 172.21.15.148:0/4032838663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9600f55b0 msgr2=0x7fe9600f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.223+0000 7fe968354700 1 --2- 172.21.15.148:0/4032838663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9600f55b0 0x7fe9600f59d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe950004660 tx=0x7fe950009b30).stop 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.223+0000 7fe968354700 1 -- 172.21.15.148:0/4032838663 shutdown_connections 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.223+0000 7fe968354700 1 --2- 172.21.15.148:0/4032838663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9600f6850 0x7fe9600f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.223+0000 7fe968354700 1 --2- 172.21.15.148:0/4032838663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9600f55b0 0x7fe9600f59d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.223+0000 7fe968354700 1 -- 172.21.15.148:0/4032838663 >> 172.21.15.148:0/4032838663 conn(0x7fe9600f0d10 msgr2=0x7fe9600f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.223+0000 7fe968354700 1 -- 172.21.15.148:0/4032838663 shutdown_connections 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.223+0000 7fe968354700 1 -- 172.21.15.148:0/4032838663 wait complete. 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.223+0000 7fe968354700 1 Processor -- start 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe968354700 1 -- start start 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe968354700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9600f55b0 0x7fe96010c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe968354700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9600f6850 0x7fe96010e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe968354700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9600602b0 con 0x7fe9600f55b0 2023-12-21T03:52:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe968354700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9600603f0 con 0x7fe9600f6850 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe9658ef700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9600f6850 0x7fe96010e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe9658ef700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9600f6850 0x7fe96010e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39738/0 (socket says 172.21.15.148:39738) 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe9658ef700 1 -- 172.21.15.148:0/2779446718 learned_addr learned my addr 172.21.15.148:0/2779446718 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe9660f0700 1 --2- 172.21.15.148:0/2779446718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9600f55b0 0x7fe96010c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe9658ef700 1 -- 172.21.15.148:0/2779446718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9600f55b0 msgr2=0x7fe96010c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe9658ef700 1 --2- 172.21.15.148:0/2779446718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9600f55b0 0x7fe96010c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe9658ef700 1 -- 172.21.15.148:0/2779446718 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe950005040 con 0x7fe9600f6850 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe9658ef700 1 --2- 172.21.15.148:0/2779446718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9600f6850 0x7fe96010e5b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe95c00c530 tx=0x7fe95c007e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe956ffd700 1 -- 172.21.15.148:0/2779446718 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe95c00f650 con 0x7fe9600f6850 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe956ffd700 1 -- 172.21.15.148:0/2779446718 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe95c00fc60 con 0x7fe9600f6850 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe968354700 1 -- 172.21.15.148:0/2779446718 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96010eaa0 con 0x7fe9600f6850 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe956ffd700 1 -- 172.21.15.148:0/2779446718 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe95c012c80 con 0x7fe9600f6850 2023-12-21T03:52:27.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.224+0000 7fe968354700 1 -- 172.21.15.148:0/2779446718 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96010ef20 con 0x7fe9600f6850 2023-12-21T03:52:27.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.225+0000 7fe968354700 1 -- 172.21.15.148:0/2779446718 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe960045ad0 con 0x7fe9600f6850 2023-12-21T03:52:27.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.226+0000 7fe956ffd700 1 -- 172.21.15.148:0/2779446718 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe95c010070 con 0x7fe9600f6850 2023-12-21T03:52:27.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.226+0000 7fe956ffd700 1 --2- 172.21.15.148:0/2779446718 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe94c05c2b0 0x7fe94c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:27.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.226+0000 7fe956ffd700 1 -- 172.21.15.148:0/2779446718 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fe95c0758c0 con 0x7fe9600f6850 2023-12-21T03:52:27.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.226+0000 7fe9660f0700 1 --2- 172.21.15.148:0/2779446718 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe94c05c2b0 0x7fe94c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:27.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.227+0000 7fe9660f0700 1 --2- 172.21.15.148:0/2779446718 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe94c05c2b0 0x7fe94c05e770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fe960060040 tx=0x7fe950009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:27.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.230+0000 7fe956ffd700 1 -- 172.21.15.148:0/2779446718 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe95c0452b0 con 0x7fe9600f6850 2023-12-21T03:52:27.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.438+0000 7fe968354700 1 -- 172.21.15.148:0/2779446718 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fe960027a00 con 0x7fe9600f6850 2023-12-21T03:52:27.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.442+0000 7fe956ffd700 1 -- 172.21.15.148:0/2779446718 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) v1 ==== 170+0+59 (secure 0 0 0) 0x7fe95c04c2f0 con 0x7fe9600f6850 2023-12-21T03:52:27.443 INFO:teuthology.orchestra.run.smithi148.stdout:[client.1] 2023-12-21T03:52:27.443 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQB7toNl3OY4GhAAvGjgwEEbn1grmRNZo1xxTQ== 2023-12-21T03:52:27.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.444+0000 7fe968354700 1 -- 172.21.15.148:0/2779446718 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe94c05c2b0 msgr2=0x7fe94c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:27.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.444+0000 7fe968354700 1 --2- 172.21.15.148:0/2779446718 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe94c05c2b0 0x7fe94c05e770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fe960060040 tx=0x7fe950009b30).stop 2023-12-21T03:52:27.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.444+0000 7fe968354700 1 -- 172.21.15.148:0/2779446718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9600f6850 msgr2=0x7fe96010e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:27.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.444+0000 7fe968354700 1 --2- 172.21.15.148:0/2779446718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9600f6850 0x7fe96010e5b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe95c00c530 tx=0x7fe95c007e20).stop 2023-12-21T03:52:27.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.444+0000 7fe968354700 1 -- 172.21.15.148:0/2779446718 shutdown_connections 2023-12-21T03:52:27.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.444+0000 7fe968354700 1 --2- 172.21.15.148:0/2779446718 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe94c05c2b0 0x7fe94c05e770 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:27.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.444+0000 7fe968354700 1 --2- 172.21.15.148:0/2779446718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9600f55b0 0x7fe96010c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:27.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.444+0000 7fe968354700 1 --2- 172.21.15.148:0/2779446718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9600f6850 0x7fe96010e5b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:27.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.444+0000 7fe968354700 1 -- 172.21.15.148:0/2779446718 >> 172.21.15.148:0/2779446718 conn(0x7fe9600f0d10 msgr2=0x7fe9600fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:27.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.444+0000 7fe968354700 1 -- 172.21.15.148:0/2779446718 shutdown_connections 2023-12-21T03:52:27.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-12-21T03:52:27.444+0000 7fe968354700 1 -- 172.21.15.148:0/2779446718 wait complete. 2023-12-21T03:52:27.735 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:25.642681+0000 mgr.smithi078.zidbgq (mgr.14162) 434 : cluster [DBG] pgmap v413: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:27.803 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-21T03:52:27.803 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-12-21T03:52:27.803 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-21T03:52:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:25.642681+0000 mgr.smithi078.zidbgq (mgr.14162) 434 : cluster [DBG] pgmap v413: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:27.844 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-21T03:52:28.077 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:28.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:27.439247+0000 mon.smithi148 (mon.1) 16 : audit [INF] from='client.? 172.21.15.148:0/2779446718' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-21T03:52:28.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:27.439774+0000 mon.smithi078 (mon.0) 489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-21T03:52:28.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:27.443127+0000 mon.smithi078 (mon.0) 490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-21T03:52:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:27.439247+0000 mon.smithi148 (mon.1) 16 : audit [INF] from='client.? 172.21.15.148:0/2779446718' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-21T03:52:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:27.439774+0000 mon.smithi078 (mon.0) 489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-21T03:52:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:27.443127+0000 mon.smithi078 (mon.0) 490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-21T03:52:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.016+0000 7ff65a457700 1 -- 172.21.15.78:0/3824818887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6540f56e0 msgr2=0x7ff6540f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.016+0000 7ff65a457700 1 --2- 172.21.15.78:0/3824818887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6540f56e0 0x7ff6540f5b00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7ff644004660 tx=0x7ff644009b30).stop 2023-12-21T03:52:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.016+0000 7ff65a457700 1 -- 172.21.15.78:0/3824818887 shutdown_connections 2023-12-21T03:52:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.016+0000 7ff65a457700 1 --2- 172.21.15.78:0/3824818887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6540f6980 0x7ff6540f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.016+0000 7ff65a457700 1 --2- 172.21.15.78:0/3824818887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6540f56e0 0x7ff6540f5b00 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.016+0000 7ff65a457700 1 -- 172.21.15.78:0/3824818887 >> 172.21.15.78:0/3824818887 conn(0x7ff6540f0e20 msgr2=0x7ff6540f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.016+0000 7ff65a457700 1 -- 172.21.15.78:0/3824818887 shutdown_connections 2023-12-21T03:52:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.016+0000 7ff65a457700 1 -- 172.21.15.78:0/3824818887 wait complete. 2023-12-21T03:52:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff65a457700 1 Processor -- start 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff65a457700 1 -- start start 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff65a457700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6540f56e0 0x7ff6541070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff65a457700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6540f6980 0x7ff6541075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff65a457700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff654107b60 con 0x7ff6540f56e0 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff65a457700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff654107ca0 con 0x7ff6540f6980 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff653fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6540f56e0 0x7ff6541070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff653fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6540f56e0 0x7ff6541070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40104/0 (socket says 172.21.15.78:40104) 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff653fff700 1 -- 172.21.15.78:0/2443030756 learned_addr learned my addr 172.21.15.78:0/2443030756 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff6537fe700 1 --2- 172.21.15.78:0/2443030756 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6540f6980 0x7ff6541075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff653fff700 1 -- 172.21.15.78:0/2443030756 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6540f6980 msgr2=0x7ff6541075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff653fff700 1 --2- 172.21.15.78:0/2443030756 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6540f6980 0x7ff6541075b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.017+0000 7ff653fff700 1 -- 172.21.15.78:0/2443030756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff644005040 con 0x7ff6540f56e0 2023-12-21T03:52:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.018+0000 7ff653fff700 1 --2- 172.21.15.78:0/2443030756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6540f56e0 0x7ff6541070c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff644004e30 tx=0x7ff644009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:29.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.018+0000 7ff650ff9700 1 -- 172.21.15.78:0/2443030756 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff644018070 con 0x7ff6540f56e0 2023-12-21T03:52:29.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.018+0000 7ff650ff9700 1 -- 172.21.15.78:0/2443030756 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff644013410 con 0x7ff6540f56e0 2023-12-21T03:52:29.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.018+0000 7ff65a457700 1 -- 172.21.15.78:0/2443030756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6541085e0 con 0x7ff6540f56e0 2023-12-21T03:52:29.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.018+0000 7ff650ff9700 1 -- 172.21.15.78:0/2443030756 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff64401de20 con 0x7ff6540f56e0 2023-12-21T03:52:29.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.018+0000 7ff65a457700 1 -- 172.21.15.78:0/2443030756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff654108b40 con 0x7ff6540f56e0 2023-12-21T03:52:29.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.019+0000 7ff65a457700 1 -- 172.21.15.78:0/2443030756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff654045ad0 con 0x7ff6540f56e0 2023-12-21T03:52:29.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.019+0000 7ff650ff9700 1 -- 172.21.15.78:0/2443030756 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff64401d420 con 0x7ff6540f56e0 2023-12-21T03:52:29.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.019+0000 7ff650ff9700 1 --2- 172.21.15.78:0/2443030756 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff63c05c2a0 0x7ff63c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:29.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.020+0000 7ff650ff9700 1 -- 172.21.15.78:0/2443030756 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff644077800 con 0x7ff6540f56e0 2023-12-21T03:52:29.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.020+0000 7ff6537fe700 1 --2- 172.21.15.78:0/2443030756 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff63c05c2a0 0x7ff63c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:29.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.020+0000 7ff6537fe700 1 --2- 172.21.15.78:0/2443030756 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff63c05c2a0 0x7ff63c05e760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff648004c40 tx=0x7ff64800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.023+0000 7ff650ff9700 1 -- 172.21.15.78:0/2443030756 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff64404a800 con 0x7ff6540f56e0 2023-12-21T03:52:29.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.173+0000 7ff65a457700 1 -- 172.21.15.78:0/2443030756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6540fdff0 con 0x7ff6540f56e0 2023-12-21T03:52:29.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.173+0000 7ff650ff9700 1 -- 172.21.15.78:0/2443030756 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10899 (secure 0 0 0) 0x7ff64404de20 con 0x7ff6540f56e0 2023-12-21T03:52:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:52:29.175 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":36,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","created":"2023-12-21T03:38:40.708805+0000","modified":"2023-12-21T03:52:24.419198+0000","last_up_change":"2023-12-21T03:52:23.413160+0000","last_in_change":"2023-12-21T03:52:08.761145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-21T03:49:24.663913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"fd284657-10ac-4999-b6e0-ecb000aa6e81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6803","nonce":3426347166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6805","nonce":3426347166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6809","nonce":3426347166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6807","nonce":3426347166}]},"public_addr":"172.21.15.78:6803/3426347166","cluster_addr":"172.21.15.78:6805/3426347166","heartbeat_back_addr":"172.21.15.78:6809/3426347166","heartbeat_front_addr":"172.21.15.78:6807/3426347166","state":["exists","up"]},{"osd":1,"uuid":"c36ef007-5c1b-4aa8-bee6-db97e9852730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6811","nonce":180184419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6813","nonce":180184419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6817","nonce":180184419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6815","nonce":180184419}]},"public_addr":"172.21.15.78:6811/180184419","cluster_addr":"172.21.15.78:6813/180184419","heartbeat_back_addr":"172.21.15.78:6817/180184419","heartbeat_front_addr":"172.21.15.78:6815/180184419","state":["exists","up"]},{"osd":2,"uuid":"5f46dcec-487c-4542-9726-78e82a92d85e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6819","nonce":56885027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6821","nonce":56885027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6825","nonce":56885027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6823","nonce":56885027}]},"public_addr":"172.21.15.78:6819/56885027","cluster_addr":"172.21.15.78:6821/56885027","heartbeat_back_addr":"172.21.15.78:6825/56885027","heartbeat_front_addr":"172.21.15.78:6823/56885027","state":["exists","up"]},{"osd":3,"uuid":"0a26620c-6292-46a1-b9d8-5228ba82bdb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6801","nonce":2694312634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6803","nonce":2694312634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6807","nonce":2694312634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6805","nonce":2694312634}]},"public_addr":"172.21.15.148:6801/2694312634","cluster_addr":"172.21.15.148:6803/2694312634","heartbeat_back_addr":"172.21.15.148:6807/2694312634","heartbeat_front_addr":"172.21.15.148:6805/2694312634","state":["exists","up"]},{"osd":4,"uuid":"cdb70954-ff3d-4fe1-a0e2-d3c162a98a7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6809","nonce":4024298721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6811","nonce":4024298721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6815","nonce":4024298721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6813","nonce":4024298721}]},"public_addr":"172.21.15.148:6809/4024298721","cluster_addr":"172.21.15.148:6811/4024298721","heartbeat_back_addr":"172.21.15.148:6815/4024298721","heartbeat_front_addr":"172.21.15.148:6813/4024298721","state":["exists","up"]},{"osd":5,"uuid":"3b89bd7a-f664-40fe-8501-4bcca0aa1298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6817","nonce":1364724132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6819","nonce":1364724132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6823","nonce":1364724132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6821","nonce":1364724132}]},"public_addr":"172.21.15.148:6817/1364724132","cluster_addr":"172.21.15.148:6819/1364724132","heartbeat_back_addr":"172.21.15.148:6823/1364724132","heartbeat_front_addr":"172.21.15.148:6821/1364724132","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:45:13.409677+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:45:26.800573+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:45:39.648051+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:49:29.356341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:52:08.560233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:52:21.548979+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2658621347":"2023-12-22T03:39:30.344015+0000","172.21.15.78:6801/1046721827":"2023-12-22T03:39:30.344015+0000","172.21.15.78:0/695730273":"2023-12-22T03:39:18.762964+0000","172.21.15.78:6800/1548863549":"2023-12-22T03:39:18.762964+0000","172.21.15.78:0/2913812525":"2023-12-22T03:39:18.762964+0000","172.21.15.78:6801/1548863549":"2023-12-22T03:39:18.762964+0000","172.21.15.78:0/3749777330":"2023-12-22T03:38:58.374051+0000","172.21.15.78:6800/1046721827":"2023-12-22T03:39:30.344015+0000","172.21.15.78:6800/3728780489":"2023-12-22T03:38:58.374051+0000","172.21.15.78:0/1459790845":"2023-12-22T03:38:58.374051+0000","172.21.15.78:0/3193560983":"2023-12-22T03:39:30.344015+0000","172.21.15.78:6801/3728780489":"2023-12-22T03:38:58.374051+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-21T03:52:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.175+0000 7ff65a457700 1 -- 172.21.15.78:0/2443030756 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff63c05c2a0 msgr2=0x7ff63c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.175+0000 7ff65a457700 1 --2- 172.21.15.78:0/2443030756 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff63c05c2a0 0x7ff63c05e760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff648004c40 tx=0x7ff64800b040).stop 2023-12-21T03:52:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.175+0000 7ff65a457700 1 -- 172.21.15.78:0/2443030756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6540f56e0 msgr2=0x7ff6541070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.175+0000 7ff65a457700 1 --2- 172.21.15.78:0/2443030756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6540f56e0 0x7ff6541070c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff644004e30 tx=0x7ff644009b30).stop 2023-12-21T03:52:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.175+0000 7ff65a457700 1 -- 172.21.15.78:0/2443030756 shutdown_connections 2023-12-21T03:52:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.175+0000 7ff65a457700 1 --2- 172.21.15.78:0/2443030756 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff63c05c2a0 0x7ff63c05e760 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.175+0000 7ff65a457700 1 --2- 172.21.15.78:0/2443030756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6540f56e0 0x7ff6541070c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.175+0000 7ff65a457700 1 --2- 172.21.15.78:0/2443030756 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6540f6980 0x7ff6541075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.175+0000 7ff65a457700 1 -- 172.21.15.78:0/2443030756 >> 172.21.15.78:0/2443030756 conn(0x7ff6540f0e20 msgr2=0x7ff6540603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.175+0000 7ff65a457700 1 -- 172.21.15.78:0/2443030756 shutdown_connections 2023-12-21T03:52:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:29.175+0000 7ff65a457700 1 -- 172.21.15.78:0/2443030756 wait complete. 2023-12-21T03:52:29.455 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:27.643081+0000 mgr.smithi078.zidbgq (mgr.14162) 435 : cluster [DBG] pgmap v414: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:29.455 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:29.174490+0000 mon.smithi078 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.78:0/2443030756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-21T03:52:29.538 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-12-21T03:49:24.663913+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2023-12-21T03:52:29.538 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd pool get device_health_metrics pg_num 2023-12-21T03:52:29.758 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:29.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:27.643081+0000 mgr.smithi078.zidbgq (mgr.14162) 435 : cluster [DBG] pgmap v414: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:29.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:29.174490+0000 mon.smithi078 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.78:0/2443030756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-21T03:52:30.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 -- 172.21.15.78:0/3935388261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0740f67e0 msgr2=0x7fe0740f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 --2- 172.21.15.78:0/3935388261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0740f67e0 0x7fe0740f6c00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fe068004660 tx=0x7fe068009b30).stop 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 -- 172.21.15.78:0/3935388261 shutdown_connections 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 --2- 172.21.15.78:0/3935388261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0740f7a80 0x7fe0740f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 --2- 172.21.15.78:0/3935388261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0740f67e0 0x7fe0740f6c00 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 -- 172.21.15.78:0/3935388261 >> 172.21.15.78:0/3935388261 conn(0x7fe0740f1f80 msgr2=0x7fe0740f43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 -- 172.21.15.78:0/3935388261 shutdown_connections 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 -- 172.21.15.78:0/3935388261 wait complete. 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 Processor -- start 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 -- start start 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0740f67e0 0x7fe07406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0740f7a80 0x7fe074069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:30.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07406a410 con 0x7fe0740f67e0 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.699+0000 7fe078f71700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07406a550 con 0x7fe0740f7a80 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe07259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0740f67e0 0x7fe07406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe07259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0740f67e0 0x7fe07406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40124/0 (socket says 172.21.15.78:40124) 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe07259c700 1 -- 172.21.15.78:0/2073915614 learned_addr learned my addr 172.21.15.78:0/2073915614 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe07259c700 1 -- 172.21.15.78:0/2073915614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0740f7a80 msgr2=0x7fe074069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe071d9b700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0740f7a80 0x7fe074069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe07259c700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0740f7a80 0x7fe074069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe07259c700 1 -- 172.21.15.78:0/2073915614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe068005040 con 0x7fe0740f67e0 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe071d9b700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0740f7a80 0x7fe074069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe07259c700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0740f67e0 0x7fe07406b9a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fe068009f50 tx=0x7fe068009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe062ffd700 1 -- 172.21.15.78:0/2073915614 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe068018070 con 0x7fe0740f67e0 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe062ffd700 1 -- 172.21.15.78:0/2073915614 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe068013410 con 0x7fe0740f67e0 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe078f71700 1 -- 172.21.15.78:0/2073915614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07406a7d0 con 0x7fe0740f67e0 2023-12-21T03:52:30.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe062ffd700 1 -- 172.21.15.78:0/2073915614 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe06801c620 con 0x7fe0740f67e0 2023-12-21T03:52:30.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.700+0000 7fe078f71700 1 -- 172.21.15.78:0/2073915614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07410f0e0 con 0x7fe0740f67e0 2023-12-21T03:52:30.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.701+0000 7fe078f71700 1 -- 172.21.15.78:0/2073915614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe074045ad0 con 0x7fe0740f67e0 2023-12-21T03:52:30.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.701+0000 7fe062ffd700 1 -- 172.21.15.78:0/2073915614 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe06801d420 con 0x7fe0740f67e0 2023-12-21T03:52:30.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.702+0000 7fe062ffd700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe05c05c2a0 0x7fe05c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:30.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.702+0000 7fe062ffd700 1 -- 172.21.15.78:0/2073915614 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fe0680779f0 con 0x7fe0740f67e0 2023-12-21T03:52:30.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.702+0000 7fe071d9b700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe05c05c2a0 0x7fe05c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:30.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.702+0000 7fe071d9b700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe05c05c2a0 0x7fe05c05e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fe06400a880 tx=0x7fe064008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:30.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.706+0000 7fe062ffd700 1 -- 172.21.15.78:0/2073915614 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe068079e60 con 0x7fe0740f67e0 2023-12-21T03:52:30.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.854+0000 7fe078f71700 1 -- 172.21.15.78:0/2073915614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe074027a00 con 0x7fe0740f67e0 2023-12-21T03:52:30.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.855+0000 7fe062ffd700 1 -- 172.21.15.78:0/2073915614 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v36) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe06804a9f0 con 0x7fe0740f67e0 2023-12-21T03:52:30.856 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-12-21T03:52:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.856+0000 7fe078f71700 1 -- 172.21.15.78:0/2073915614 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe05c05c2a0 msgr2=0x7fe05c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.856+0000 7fe078f71700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe05c05c2a0 0x7fe05c05e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fe06400a880 tx=0x7fe064008040).stop 2023-12-21T03:52:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.856+0000 7fe078f71700 1 -- 172.21.15.78:0/2073915614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0740f67e0 msgr2=0x7fe07406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.856+0000 7fe078f71700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0740f67e0 0x7fe07406b9a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fe068009f50 tx=0x7fe068009b30).stop 2023-12-21T03:52:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.857+0000 7fe078f71700 1 -- 172.21.15.78:0/2073915614 shutdown_connections 2023-12-21T03:52:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.857+0000 7fe078f71700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fe05c05c2a0 0x7fe05c05e760 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.857+0000 7fe078f71700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0740f67e0 0x7fe07406b9a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.857+0000 7fe078f71700 1 --2- 172.21.15.78:0/2073915614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0740f7a80 0x7fe074069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.857+0000 7fe078f71700 1 -- 172.21.15.78:0/2073915614 >> 172.21.15.78:0/2073915614 conn(0x7fe0740f1f80 msgr2=0x7fe0740fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.857+0000 7fe078f71700 1 -- 172.21.15.78:0/2073915614 shutdown_connections 2023-12-21T03:52:30.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:30.857+0000 7fe078f71700 1 -- 172.21.15.78:0/2073915614 wait complete. 2023-12-21T03:52:31.222 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-21T03:52:31.222 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-12-21T03:52:31.222 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph mgr dump --format=json 2023-12-21T03:52:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:29.643419+0000 mgr.smithi078.zidbgq (mgr.14162) 436 : cluster [DBG] pgmap v415: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:30.565486+0000 mon.smithi078 (mon.0) 492 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:52:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:30.565670+0000 mon.smithi078 (mon.0) 493 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:52:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:30.856339+0000 mon.smithi078 (mon.0) 494 : audit [DBG] from='client.? 172.21.15.78:0/2073915614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-12-21T03:52:31.789 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:29.643419+0000 mgr.smithi078.zidbgq (mgr.14162) 436 : cluster [DBG] pgmap v415: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:31.789 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:30.565486+0000 mon.smithi078 (mon.0) 492 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:52:31.789 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:30.565670+0000 mon.smithi078 (mon.0) 493 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:52:31.789 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:30.856339+0000 mon.smithi078 (mon.0) 494 : audit [DBG] from='client.? 172.21.15.78:0/2073915614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-12-21T03:52:32.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.433+0000 7f8d5ab64700 1 -- 172.21.15.78:0/515129802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d540f6c90 msgr2=0x7f8d540f70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:32.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.433+0000 7f8d5ab64700 1 --2- 172.21.15.78:0/515129802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d540f6c90 0x7f8d540f70b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f8d48004660 tx=0x7f8d48009b30).stop 2023-12-21T03:52:32.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.433+0000 7f8d5ab64700 1 -- 172.21.15.78:0/515129802 shutdown_connections 2023-12-21T03:52:32.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.433+0000 7f8d5ab64700 1 --2- 172.21.15.78:0/515129802 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d540f7630 0x7f8d540f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:32.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.433+0000 7f8d5ab64700 1 --2- 172.21.15.78:0/515129802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d540f6c90 0x7f8d540f70b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:32.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.433+0000 7f8d5ab64700 1 -- 172.21.15.78:0/515129802 >> 172.21.15.78:0/515129802 conn(0x7f8d540f0590 msgr2=0x7f8d540f29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:32.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.433+0000 7f8d5ab64700 1 -- 172.21.15.78:0/515129802 shutdown_connections 2023-12-21T03:52:32.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.433+0000 7f8d5ab64700 1 -- 172.21.15.78:0/515129802 wait complete. 2023-12-21T03:52:32.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d5ab64700 1 Processor -- start 2023-12-21T03:52:32.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d5ab64700 1 -- start start 2023-12-21T03:52:32.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d5ab64700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d540f6c90 0x7f8d54109e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:32.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d5ab64700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d540f7630 0x7f8d5410c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d5ab64700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d540f5730 con 0x7f8d540f7630 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d5ab64700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d540f5870 con 0x7f8d540f6c90 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d53fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d540f7630 0x7f8d5410c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d53fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d540f7630 0x7f8d5410c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40146/0 (socket says 172.21.15.78:40146) 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d53fff700 1 -- 172.21.15.78:0/3223937805 learned_addr learned my addr 172.21.15.78:0/3223937805 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d58900700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d540f6c90 0x7f8d54109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d53fff700 1 -- 172.21.15.78:0/3223937805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d540f6c90 msgr2=0x7f8d54109e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d53fff700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d540f6c90 0x7f8d54109e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d53fff700 1 -- 172.21.15.78:0/3223937805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d48005040 con 0x7f8d540f7630 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.434+0000 7f8d58900700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d540f6c90 0x7f8d54109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.435+0000 7f8d53fff700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d540f7630 0x7f8d5410c300 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f8d4000c790 tx=0x7f8d40007d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.435+0000 7f8d517fa700 1 -- 172.21.15.78:0/3223937805 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d4000e680 con 0x7f8d540f7630 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.435+0000 7f8d517fa700 1 -- 172.21.15.78:0/3223937805 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d4000ec90 con 0x7f8d540f7630 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.435+0000 7f8d5ab64700 1 -- 172.21.15.78:0/3223937805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5410c8b0 con 0x7f8d540f7630 2023-12-21T03:52:32.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.435+0000 7f8d517fa700 1 -- 172.21.15.78:0/3223937805 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d40013cd0 con 0x7f8d540f7630 2023-12-21T03:52:32.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.435+0000 7f8d5ab64700 1 -- 172.21.15.78:0/3223937805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5410ce70 con 0x7f8d540f7630 2023-12-21T03:52:32.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.436+0000 7f8d5ab64700 1 -- 172.21.15.78:0/3223937805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d54045ad0 con 0x7f8d540f7630 2023-12-21T03:52:32.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.436+0000 7f8d517fa700 1 -- 172.21.15.78:0/3223937805 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8d4001a060 con 0x7f8d540f7630 2023-12-21T03:52:32.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.436+0000 7f8d517fa700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4405c2a0 0x7f8d4405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:32.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.436+0000 7f8d517fa700 1 -- 172.21.15.78:0/3223937805 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8d40075ce0 con 0x7f8d540f7630 2023-12-21T03:52:32.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.436+0000 7f8d58900700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4405c2a0 0x7f8d4405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:32.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.437+0000 7f8d58900700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4405c2a0 0x7f8d4405e760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8d540f53b0 tx=0x7f8d48009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:32.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.440+0000 7f8d517fa700 1 -- 172.21.15.78:0/3223937805 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d40048d00 con 0x7f8d540f7630 2023-12-21T03:52:32.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.628+0000 7f8d5ab64700 1 -- 172.21.15.78:0/3223937805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f8d54027a00 con 0x7f8d540f7630 2023-12-21T03:52:32.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.631+0000 7f8d517fa700 1 -- 172.21.15.78:0/3223937805 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v18) v1 ==== 74+0+143500 (secure 0 0 0) 0x7f8d4004c320 con 0x7f8d540f7630 2023-12-21T03:52:32.634 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:52:32.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.633+0000 7f8d5ab64700 1 -- 172.21.15.78:0/3223937805 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4405c2a0 msgr2=0x7f8d4405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:32.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.633+0000 7f8d5ab64700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4405c2a0 0x7f8d4405e760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8d540f53b0 tx=0x7f8d48009b30).stop 2023-12-21T03:52:32.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.633+0000 7f8d5ab64700 1 -- 172.21.15.78:0/3223937805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d540f7630 msgr2=0x7f8d5410c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:32.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.633+0000 7f8d5ab64700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d540f7630 0x7f8d5410c300 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f8d4000c790 tx=0x7f8d40007d40).stop 2023-12-21T03:52:32.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.633+0000 7f8d5ab64700 1 -- 172.21.15.78:0/3223937805 shutdown_connections 2023-12-21T03:52:32.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.633+0000 7f8d5ab64700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8d4405c2a0 0x7f8d4405e760 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:32.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.633+0000 7f8d5ab64700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d540f6c90 0x7f8d54109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:32.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.633+0000 7f8d5ab64700 1 --2- 172.21.15.78:0/3223937805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d540f7630 0x7f8d5410c300 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:32.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.633+0000 7f8d5ab64700 1 -- 172.21.15.78:0/3223937805 >> 172.21.15.78:0/3223937805 conn(0x7f8d540f0590 msgr2=0x7f8d540fa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:32.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.633+0000 7f8d5ab64700 1 -- 172.21.15.78:0/3223937805 shutdown_connections 2023-12-21T03:52:32.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:32.633+0000 7f8d5ab64700 1 -- 172.21.15.78:0/3223937805 wait complete. 2023-12-21T03:52:33.005 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":18,"active_gid":14162,"active_name":"smithi078.zidbgq","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":2487427889},{"type":"v1","addr":"172.21.15.78:6801","nonce":2487427889}]},"active_addr":"172.21.15.78:6801/2487427889","active_change":"2023-12-21T03:39:30.344389+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":14384,"name":"smithi148.tgpyxd","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi078.front.sepia.ceph.com:8443/","prometheus":"http://smithi078.front.sepia.ceph.com:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":4,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2346179867}]},{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2682456583}]}]}} 2023-12-21T03:52:33.008 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-12-21T03:52:33.008 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-12-21T03:52:33.008 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-21T03:52:33.228 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:33.734 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:31.643810+0000 mgr.smithi078.zidbgq (mgr.14162) 437 : cluster [DBG] pgmap v416: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:33.734 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:32.629903+0000 mon.smithi078 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.78:0/3223937805' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-21T03:52:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:31.643810+0000 mgr.smithi078.zidbgq (mgr.14162) 437 : cluster [DBG] pgmap v416: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:32.629903+0000 mon.smithi078 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.78:0/3223937805' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-21T03:52:34.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.229+0000 7faf8a147700 1 -- 172.21.15.78:0/1438365830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf840f67c0 msgr2=0x7faf840f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.229+0000 7faf8a147700 1 --2- 172.21.15.78:0/1438365830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf840f67c0 0x7faf840f6be0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7faf6c004660 tx=0x7faf6c009b30).stop 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.229+0000 7faf8a147700 1 -- 172.21.15.78:0/1438365830 shutdown_connections 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.229+0000 7faf8a147700 1 --2- 172.21.15.78:0/1438365830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faf840f7a60 0x7faf840f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.229+0000 7faf8a147700 1 --2- 172.21.15.78:0/1438365830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf840f67c0 0x7faf840f6be0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.229+0000 7faf8a147700 1 -- 172.21.15.78:0/1438365830 >> 172.21.15.78:0/1438365830 conn(0x7faf840f1f20 msgr2=0x7faf840f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.229+0000 7faf8a147700 1 -- 172.21.15.78:0/1438365830 shutdown_connections 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.229+0000 7faf8a147700 1 -- 172.21.15.78:0/1438365830 wait complete. 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.229+0000 7faf8a147700 1 Processor -- start 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.229+0000 7faf8a147700 1 -- start start 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf8a147700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faf840f67c0 0x7faf841070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf8a147700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf840f7a60 0x7faf84107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf8a147700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf84107b40 con 0x7faf840f7a60 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf8a147700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf84107c80 con 0x7faf840f67c0 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf82ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf840f7a60 0x7faf84107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf82ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf840f7a60 0x7faf84107590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40172/0 (socket says 172.21.15.78:40172) 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf82ffd700 1 -- 172.21.15.78:0/1933305969 learned_addr learned my addr 172.21.15.78:0/1933305969 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf837fe700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faf840f67c0 0x7faf841070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf82ffd700 1 -- 172.21.15.78:0/1933305969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faf840f67c0 msgr2=0x7faf841070a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf82ffd700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faf840f67c0 0x7faf841070a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf82ffd700 1 -- 172.21.15.78:0/1933305969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c005040 con 0x7faf840f7a60 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf837fe700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faf840f67c0 0x7faf841070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf82ffd700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf840f7a60 0x7faf84107590 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7faf740026d0 tx=0x7faf74009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf7bfff700 1 -- 172.21.15.78:0/1933305969 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf74009320 con 0x7faf840f7a60 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf8a147700 1 -- 172.21.15.78:0/1933305969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf84108620 con 0x7faf840f7a60 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf7bfff700 1 -- 172.21.15.78:0/1933305969 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf7400f040 con 0x7faf840f7a60 2023-12-21T03:52:34.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf7bfff700 1 -- 172.21.15.78:0/1933305969 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf740147e0 con 0x7faf840f7a60 2023-12-21T03:52:34.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.230+0000 7faf8a147700 1 -- 172.21.15.78:0/1933305969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf84108be0 con 0x7faf840f7a60 2023-12-21T03:52:34.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.231+0000 7faf8a147700 1 -- 172.21.15.78:0/1933305969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf84045ad0 con 0x7faf840f7a60 2023-12-21T03:52:34.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.231+0000 7faf7bfff700 1 -- 172.21.15.78:0/1933305969 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7faf74009480 con 0x7faf840f7a60 2023-12-21T03:52:34.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.232+0000 7faf7bfff700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faf7005c2a0 0x7faf7005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:34.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.232+0000 7faf7bfff700 1 -- 172.21.15.78:0/1933305969 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7faf74076270 con 0x7faf840f7a60 2023-12-21T03:52:34.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.232+0000 7faf837fe700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faf7005c2a0 0x7faf7005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:34.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.232+0000 7faf837fe700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faf7005c2a0 0x7faf7005e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7faf6c004e30 tx=0x7faf6c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:34.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.236+0000 7faf7bfff700 1 -- 172.21.15.78:0/1933305969 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf74049270 con 0x7faf840f7a60 2023-12-21T03:52:34.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.387+0000 7faf8a147700 1 -- 172.21.15.78:0/1933305969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf84027a00 con 0x7faf840f7a60 2023-12-21T03:52:34.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.388+0000 7faf7bfff700 1 -- 172.21.15.78:0/1933305969 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10899 (secure 0 0 0) 0x7faf7404c890 con 0x7faf840f7a60 2023-12-21T03:52:34.390 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:52:34.390 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":36,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","created":"2023-12-21T03:38:40.708805+0000","modified":"2023-12-21T03:52:24.419198+0000","last_up_change":"2023-12-21T03:52:23.413160+0000","last_in_change":"2023-12-21T03:52:08.761145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-21T03:49:24.663913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"fd284657-10ac-4999-b6e0-ecb000aa6e81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6803","nonce":3426347166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6805","nonce":3426347166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6809","nonce":3426347166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6807","nonce":3426347166}]},"public_addr":"172.21.15.78:6803/3426347166","cluster_addr":"172.21.15.78:6805/3426347166","heartbeat_back_addr":"172.21.15.78:6809/3426347166","heartbeat_front_addr":"172.21.15.78:6807/3426347166","state":["exists","up"]},{"osd":1,"uuid":"c36ef007-5c1b-4aa8-bee6-db97e9852730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6811","nonce":180184419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6813","nonce":180184419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6817","nonce":180184419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6815","nonce":180184419}]},"public_addr":"172.21.15.78:6811/180184419","cluster_addr":"172.21.15.78:6813/180184419","heartbeat_back_addr":"172.21.15.78:6817/180184419","heartbeat_front_addr":"172.21.15.78:6815/180184419","state":["exists","up"]},{"osd":2,"uuid":"5f46dcec-487c-4542-9726-78e82a92d85e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6819","nonce":56885027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6821","nonce":56885027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6825","nonce":56885027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6823","nonce":56885027}]},"public_addr":"172.21.15.78:6819/56885027","cluster_addr":"172.21.15.78:6821/56885027","heartbeat_back_addr":"172.21.15.78:6825/56885027","heartbeat_front_addr":"172.21.15.78:6823/56885027","state":["exists","up"]},{"osd":3,"uuid":"0a26620c-6292-46a1-b9d8-5228ba82bdb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6801","nonce":2694312634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6803","nonce":2694312634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6807","nonce":2694312634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6805","nonce":2694312634}]},"public_addr":"172.21.15.148:6801/2694312634","cluster_addr":"172.21.15.148:6803/2694312634","heartbeat_back_addr":"172.21.15.148:6807/2694312634","heartbeat_front_addr":"172.21.15.148:6805/2694312634","state":["exists","up"]},{"osd":4,"uuid":"cdb70954-ff3d-4fe1-a0e2-d3c162a98a7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6809","nonce":4024298721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6811","nonce":4024298721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6815","nonce":4024298721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6813","nonce":4024298721}]},"public_addr":"172.21.15.148:6809/4024298721","cluster_addr":"172.21.15.148:6811/4024298721","heartbeat_back_addr":"172.21.15.148:6815/4024298721","heartbeat_front_addr":"172.21.15.148:6813/4024298721","state":["exists","up"]},{"osd":5,"uuid":"3b89bd7a-f664-40fe-8501-4bcca0aa1298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6817","nonce":1364724132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6819","nonce":1364724132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6823","nonce":1364724132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6821","nonce":1364724132}]},"public_addr":"172.21.15.148:6817/1364724132","cluster_addr":"172.21.15.148:6819/1364724132","heartbeat_back_addr":"172.21.15.148:6823/1364724132","heartbeat_front_addr":"172.21.15.148:6821/1364724132","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:45:13.409677+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:45:26.800573+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:45:39.648051+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:49:29.356341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:52:08.560233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:52:21.548979+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2658621347":"2023-12-22T03:39:30.344015+0000","172.21.15.78:6801/1046721827":"2023-12-22T03:39:30.344015+0000","172.21.15.78:0/695730273":"2023-12-22T03:39:18.762964+0000","172.21.15.78:6800/1548863549":"2023-12-22T03:39:18.762964+0000","172.21.15.78:0/2913812525":"2023-12-22T03:39:18.762964+0000","172.21.15.78:6801/1548863549":"2023-12-22T03:39:18.762964+0000","172.21.15.78:0/3749777330":"2023-12-22T03:38:58.374051+0000","172.21.15.78:6800/1046721827":"2023-12-22T03:39:30.344015+0000","172.21.15.78:6800/3728780489":"2023-12-22T03:38:58.374051+0000","172.21.15.78:0/1459790845":"2023-12-22T03:38:58.374051+0000","172.21.15.78:0/3193560983":"2023-12-22T03:39:30.344015+0000","172.21.15.78:6801/3728780489":"2023-12-22T03:38:58.374051+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-21T03:52:34.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.390+0000 7faf8a147700 1 -- 172.21.15.78:0/1933305969 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faf7005c2a0 msgr2=0x7faf7005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:34.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.390+0000 7faf8a147700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faf7005c2a0 0x7faf7005e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7faf6c004e30 tx=0x7faf6c009b30).stop 2023-12-21T03:52:34.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.390+0000 7faf8a147700 1 -- 172.21.15.78:0/1933305969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf840f7a60 msgr2=0x7faf84107590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:34.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.390+0000 7faf8a147700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf840f7a60 0x7faf84107590 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7faf740026d0 tx=0x7faf74009de0).stop 2023-12-21T03:52:34.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.390+0000 7faf8a147700 1 -- 172.21.15.78:0/1933305969 shutdown_connections 2023-12-21T03:52:34.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.390+0000 7faf8a147700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faf7005c2a0 0x7faf7005e760 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:34.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.390+0000 7faf8a147700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faf840f67c0 0x7faf841070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:34.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.390+0000 7faf8a147700 1 --2- 172.21.15.78:0/1933305969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf840f7a60 0x7faf84107590 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:34.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.390+0000 7faf8a147700 1 -- 172.21.15.78:0/1933305969 >> 172.21.15.78:0/1933305969 conn(0x7faf840f1f20 msgr2=0x7faf840fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:34.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.390+0000 7faf8a147700 1 -- 172.21.15.78:0/1933305969 shutdown_connections 2023-12-21T03:52:34.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:34.390+0000 7faf8a147700 1 -- 172.21.15.78:0/1933305969 wait complete. 2023-12-21T03:52:34.746 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-12-21T03:52:34.747 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-21T03:52:34.965 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:35.701 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:33.644279+0000 mgr.smithi078.zidbgq (mgr.14162) 438 : cluster [DBG] pgmap v417: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:35.701 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:34.389333+0000 mon.smithi078 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.78:0/1933305969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-21T03:52:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:33.644279+0000 mgr.smithi078.zidbgq (mgr.14162) 438 : cluster [DBG] pgmap v417: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:34.389333+0000 mon.smithi078 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.78:0/1933305969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-21T03:52:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.877+0000 7f334c055700 1 -- 172.21.15.78:0/307490557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3344060830 msgr2=0x7f33440f93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.877+0000 7f334c055700 1 --2- 172.21.15.78:0/307490557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3344060830 0x7f33440f93e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f3334004660 tx=0x7f3334009b30).stop 2023-12-21T03:52:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.877+0000 7f334c055700 1 -- 172.21.15.78:0/307490557 shutdown_connections 2023-12-21T03:52:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.877+0000 7f334c055700 1 --2- 172.21.15.78:0/307490557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33440f9960 0x7f33440fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:52:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.877+0000 7f334c055700 1 --2- 172.21.15.78:0/307490557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3344060830 0x7f33440f93e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.877+0000 7f334c055700 1 -- 172.21.15.78:0/307490557 >> 172.21.15.78:0/307490557 conn(0x7f33440f0e20 msgr2=0x7f33440f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.877+0000 7f334c055700 1 -- 172.21.15.78:0/307490557 shutdown_connections 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.877+0000 7f334c055700 1 -- 172.21.15.78:0/307490557 wait complete. 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f334c055700 1 Processor -- start 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f334c055700 1 -- start start 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f334c055700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3344060830 0x7f33441070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f334c055700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33440f9960 0x7f33441075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f334c055700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3344107bf0 con 0x7f3344060830 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f334c055700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33441084a0 con 0x7f33440f9960 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f3349df1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3344060830 0x7f33441070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f3349df1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3344060830 0x7f33441070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40200/0 (socket says 172.21.15.78:40200) 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f3349df1700 1 -- 172.21.15.78:0/3898629263 learned_addr learned my addr 172.21.15.78:0/3898629263 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f33495f0700 1 --2- 172.21.15.78:0/3898629263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33440f9960 0x7f33441075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f3349df1700 1 -- 172.21.15.78:0/3898629263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33440f9960 msgr2=0x7f33441075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f3349df1700 1 --2- 172.21.15.78:0/3898629263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33440f9960 0x7f33441075b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.878+0000 7f3349df1700 1 -- 172.21.15.78:0/3898629263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334005040 con 0x7f3344060830 2023-12-21T03:52:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.879+0000 7f3349df1700 1 --2- 172.21.15.78:0/3898629263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3344060830 0x7f33441070c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f3334009830 tx=0x7f3334009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.879+0000 7f333a7fc700 1 -- 172.21.15.78:0/3898629263 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3334018070 con 0x7f3344060830 2023-12-21T03:52:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.879+0000 7f333a7fc700 1 -- 172.21.15.78:0/3898629263 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3334013410 con 0x7f3344060830 2023-12-21T03:52:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.879+0000 7f334c055700 1 -- 172.21.15.78:0/3898629263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3344108720 con 0x7f3344060830 2023-12-21T03:52:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.879+0000 7f333a7fc700 1 -- 172.21.15.78:0/3898629263 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f333401c620 con 0x7f3344060830 2023-12-21T03:52:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.879+0000 7f334c055700 1 -- 172.21.15.78:0/3898629263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3344108c80 con 0x7f3344060830 2023-12-21T03:52:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.880+0000 7f334c055700 1 -- 172.21.15.78:0/3898629263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3344059100 con 0x7f3344060830 2023-12-21T03:52:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.880+0000 7f333a7fc700 1 -- 172.21.15.78:0/3898629263 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f333401c8e0 con 0x7f3344060830 2023-12-21T03:52:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.880+0000 7f333a7fc700 1 --2- 172.21.15.78:0/3898629263 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f333005c2a0 0x7f333005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.880+0000 7f333a7fc700 1 -- 172.21.15.78:0/3898629263 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3334077a60 con 0x7f3344060830 2023-12-21T03:52:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.880+0000 7f33495f0700 1 --2- 172.21.15.78:0/3898629263 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f333005c2a0 0x7f333005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.881+0000 7f33495f0700 1 --2- 172.21.15.78:0/3898629263 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f333005c2a0 0x7f333005e760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f3344060600 tx=0x7f3340008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:35.884+0000 7f333a7fc700 1 -- 172.21.15.78:0/3898629263 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f333404aa00 con 0x7f3344060830 2023-12-21T03:52:36.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.034+0000 7f334c055700 1 -- 172.21.15.78:0/3898629263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3344045ad0 con 0x7f3344060830 2023-12-21T03:52:36.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.034+0000 7f333a7fc700 1 -- 172.21.15.78:0/3898629263 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10899 (secure 0 0 0) 0x7f333404e020 con 0x7f3344060830 2023-12-21T03:52:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:52:36.036 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":36,"fsid":"5ef0788c-9fb2-11ee-95a5-87774f69a715","created":"2023-12-21T03:38:40.708805+0000","modified":"2023-12-21T03:52:24.419198+0000","last_up_change":"2023-12-21T03:52:23.413160+0000","last_in_change":"2023-12-21T03:52:08.761145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-21T03:49:24.663913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"fd284657-10ac-4999-b6e0-ecb000aa6e81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6803","nonce":3426347166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6805","nonce":3426347166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6809","nonce":3426347166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3426347166},{"type":"v1","addr":"172.21.15.78:6807","nonce":3426347166}]},"public_addr":"172.21.15.78:6803/3426347166","cluster_addr":"172.21.15.78:6805/3426347166","heartbeat_back_addr":"172.21.15.78:6809/3426347166","heartbeat_front_addr":"172.21.15.78:6807/3426347166","state":["exists","up"]},{"osd":1,"uuid":"c36ef007-5c1b-4aa8-bee6-db97e9852730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6811","nonce":180184419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6813","nonce":180184419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6817","nonce":180184419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":180184419},{"type":"v1","addr":"172.21.15.78:6815","nonce":180184419}]},"public_addr":"172.21.15.78:6811/180184419","cluster_addr":"172.21.15.78:6813/180184419","heartbeat_back_addr":"172.21.15.78:6817/180184419","heartbeat_front_addr":"172.21.15.78:6815/180184419","state":["exists","up"]},{"osd":2,"uuid":"5f46dcec-487c-4542-9726-78e82a92d85e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6819","nonce":56885027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6821","nonce":56885027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6825","nonce":56885027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":56885027},{"type":"v1","addr":"172.21.15.78:6823","nonce":56885027}]},"public_addr":"172.21.15.78:6819/56885027","cluster_addr":"172.21.15.78:6821/56885027","heartbeat_back_addr":"172.21.15.78:6825/56885027","heartbeat_front_addr":"172.21.15.78:6823/56885027","state":["exists","up"]},{"osd":3,"uuid":"0a26620c-6292-46a1-b9d8-5228ba82bdb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6801","nonce":2694312634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6803","nonce":2694312634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6807","nonce":2694312634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":2694312634},{"type":"v1","addr":"172.21.15.148:6805","nonce":2694312634}]},"public_addr":"172.21.15.148:6801/2694312634","cluster_addr":"172.21.15.148:6803/2694312634","heartbeat_back_addr":"172.21.15.148:6807/2694312634","heartbeat_front_addr":"172.21.15.148:6805/2694312634","state":["exists","up"]},{"osd":4,"uuid":"cdb70954-ff3d-4fe1-a0e2-d3c162a98a7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6809","nonce":4024298721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6811","nonce":4024298721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6815","nonce":4024298721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":4024298721},{"type":"v1","addr":"172.21.15.148:6813","nonce":4024298721}]},"public_addr":"172.21.15.148:6809/4024298721","cluster_addr":"172.21.15.148:6811/4024298721","heartbeat_back_addr":"172.21.15.148:6815/4024298721","heartbeat_front_addr":"172.21.15.148:6813/4024298721","state":["exists","up"]},{"osd":5,"uuid":"3b89bd7a-f664-40fe-8501-4bcca0aa1298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6817","nonce":1364724132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6819","nonce":1364724132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6823","nonce":1364724132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1364724132},{"type":"v1","addr":"172.21.15.148:6821","nonce":1364724132}]},"public_addr":"172.21.15.148:6817/1364724132","cluster_addr":"172.21.15.148:6819/1364724132","heartbeat_back_addr":"172.21.15.148:6823/1364724132","heartbeat_front_addr":"172.21.15.148:6821/1364724132","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:45:13.409677+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:45:26.800573+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:45:39.648051+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:49:29.356341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:52:08.560233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-12-21T03:52:21.548979+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2658621347":"2023-12-22T03:39:30.344015+0000","172.21.15.78:6801/1046721827":"2023-12-22T03:39:30.344015+0000","172.21.15.78:0/695730273":"2023-12-22T03:39:18.762964+0000","172.21.15.78:6800/1548863549":"2023-12-22T03:39:18.762964+0000","172.21.15.78:0/2913812525":"2023-12-22T03:39:18.762964+0000","172.21.15.78:6801/1548863549":"2023-12-22T03:39:18.762964+0000","172.21.15.78:0/3749777330":"2023-12-22T03:38:58.374051+0000","172.21.15.78:6800/1046721827":"2023-12-22T03:39:30.344015+0000","172.21.15.78:6800/3728780489":"2023-12-22T03:38:58.374051+0000","172.21.15.78:0/1459790845":"2023-12-22T03:38:58.374051+0000","172.21.15.78:0/3193560983":"2023-12-22T03:39:30.344015+0000","172.21.15.78:6801/3728780489":"2023-12-22T03:38:58.374051+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-21T03:52:36.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.036+0000 7f334c055700 1 -- 172.21.15.78:0/3898629263 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f333005c2a0 msgr2=0x7f333005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:36.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.036+0000 7f334c055700 1 --2- 172.21.15.78:0/3898629263 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f333005c2a0 0x7f333005e760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f3344060600 tx=0x7f3340008040).stop 2023-12-21T03:52:36.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.036+0000 7f334c055700 1 -- 172.21.15.78:0/3898629263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3344060830 msgr2=0x7f33441070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:36.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.036+0000 7f334c055700 1 --2- 172.21.15.78:0/3898629263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3344060830 0x7f33441070c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f3334009830 tx=0x7f3334009b30).stop 2023-12-21T03:52:36.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.036+0000 7f334c055700 1 -- 172.21.15.78:0/3898629263 shutdown_connections 2023-12-21T03:52:36.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.036+0000 7f334c055700 1 --2- 172.21.15.78:0/3898629263 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f333005c2a0 0x7f333005e760 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:36.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.036+0000 7f334c055700 1 --2- 172.21.15.78:0/3898629263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3344060830 0x7f33441070c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:36.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.036+0000 7f334c055700 1 --2- 172.21.15.78:0/3898629263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33440f9960 0x7f33441075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:36.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.036+0000 7f334c055700 1 -- 172.21.15.78:0/3898629263 >> 172.21.15.78:0/3898629263 conn(0x7f33440f0e20 msgr2=0x7f33440f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:36.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.036+0000 7f334c055700 1 -- 172.21.15.78:0/3898629263 shutdown_connections 2023-12-21T03:52:36.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:36.036+0000 7f334c055700 1 -- 172.21.15.78:0/3898629263 wait complete. 2023-12-21T03:52:36.385 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2023-12-21T03:52:36.605 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:36.724 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:36 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:36.035459+0000 mon.smithi078 (mon.0) 497 : audit [DBG] from='client.? 172.21.15.78:0/3898629263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-21T03:52:36.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:36 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:36.035459+0000 mon.smithi078 (mon.0) 497 : audit [DBG] from='client.? 172.21.15.78:0/3898629263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-21T03:52:37.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3606404799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd600f9920 msgr2=0x7fbd600fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:37.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 --2- 172.21.15.78:0/3606404799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd600f9920 0x7fbd600fbd70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fbd50004660 tx=0x7fbd50009b30).stop 2023-12-21T03:52:37.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3606404799 shutdown_connections 2023-12-21T03:52:37.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 --2- 172.21.15.78:0/3606404799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd600f9920 0x7fbd600fbd70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:37.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 --2- 172.21.15.78:0/3606404799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd60060870 0x7fbd600f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:37.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3606404799 >> 172.21.15.78:0/3606404799 conn(0x7fbd600f0e20 msgr2=0x7fbd600f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:37.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3606404799 shutdown_connections 2023-12-21T03:52:37.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3606404799 wait complete. 2023-12-21T03:52:37.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 Processor -- start 2023-12-21T03:52:37.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 -- start start 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd60060870 0x7fbd600f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd600f9920 0x7fbd600f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600f5800 con 0x7fbd60060870 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.554+0000 7fbd64eaf700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600f5940 con 0x7fbd600f9920 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd5e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd60060870 0x7fbd600f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd5e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd60060870 0x7fbd600f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40212/0 (socket says 172.21.15.78:40212) 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd5e59c700 1 -- 172.21.15.78:0/3870732981 learned_addr learned my addr 172.21.15.78:0/3870732981 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd5dd9b700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd600f9920 0x7fbd600f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd5e59c700 1 -- 172.21.15.78:0/3870732981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd600f9920 msgr2=0x7fbd600f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd5e59c700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd600f9920 0x7fbd600f52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd5e59c700 1 -- 172.21.15.78:0/3870732981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd50005040 con 0x7fbd60060870 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd5dd9b700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd600f9920 0x7fbd600f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd5e59c700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd60060870 0x7fbd600f6e20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fbd540026d0 tx=0x7fbd54009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:37.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd4effd700 1 -- 172.21.15.78:0/3870732981 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd54009320 con 0x7fbd60060870 2023-12-21T03:52:37.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd4effd700 1 -- 172.21.15.78:0/3870732981 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd5400f040 con 0x7fbd60060870 2023-12-21T03:52:37.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3870732981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd600f5c20 con 0x7fbd60060870 2023-12-21T03:52:37.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd4effd700 1 -- 172.21.15.78:0/3870732981 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd540147e0 con 0x7fbd60060870 2023-12-21T03:52:37.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.555+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3870732981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6010f130 con 0x7fbd60060870 2023-12-21T03:52:37.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.556+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3870732981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbd6005dd30 con 0x7fbd60060870 2023-12-21T03:52:37.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.556+0000 7fbd4effd700 1 -- 172.21.15.78:0/3870732981 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbd54009480 con 0x7fbd60060870 2023-12-21T03:52:37.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.557+0000 7fbd4effd700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbd4805c2a0 0x7fbd4805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:37.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.557+0000 7fbd4effd700 1 -- 172.21.15.78:0/3870732981 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbd54076270 con 0x7fbd60060870 2023-12-21T03:52:37.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.557+0000 7fbd5dd9b700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbd4805c2a0 0x7fbd4805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:37.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.557+0000 7fbd4effd700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166] conn(0x7fbd48062030 0x7fbd48064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:37.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.557+0000 7fbd4effd700 1 -- 172.21.15.78:0/3870732981 --> [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd48064b10 con 0x7fbd48062030 2023-12-21T03:52:37.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.557+0000 7fbd4effd700 1 -- 172.21.15.78:0/3870732981 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd540788e0 con 0x7fbd60060870 2023-12-21T03:52:37.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.557+0000 7fbd5ed9d700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166] conn(0x7fbd48062030 0x7fbd48064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:37.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.557+0000 7fbd5dd9b700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbd4805c2a0 0x7fbd4805e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fbd50004e30 tx=0x7fbd50009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:37.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.557+0000 7fbd5ed9d700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166] conn(0x7fbd48062030 0x7fbd48064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:37.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.558+0000 7fbd4effd700 1 -- 172.21.15.78:0/3870732981 <== osd.0 v2:172.21.15.78:6802/3426347166 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fbd48064b10 con 0x7fbd48062030 2023-12-21T03:52:37.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3870732981 --> [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbd600581d0 con 0x7fbd48062030 2023-12-21T03:52:37.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd4effd700 1 -- 172.21.15.78:0/3870732981 <== osd.0 v2:172.21.15.78:6802/3426347166 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbd600581d0 con 0x7fbd48062030 2023-12-21T03:52:37.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166] conn(0x7fbd48062030 msgr2=0x7fbd48064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:37.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166] conn(0x7fbd48062030 0x7fbd48064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:37.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbd4805c2a0 msgr2=0x7fbd4805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:37.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbd4805c2a0 0x7fbd4805e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fbd50004e30 tx=0x7fbd50009b30).stop 2023-12-21T03:52:37.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd60060870 msgr2=0x7fbd600f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:37.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd60060870 0x7fbd600f6e20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fbd540026d0 tx=0x7fbd54009de0).stop 2023-12-21T03:52:37.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3870732981 shutdown_connections 2023-12-21T03:52:37.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6802/3426347166,v1:172.21.15.78:6803/3426347166] conn(0x7fbd48062030 0x7fbd48064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:37.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbd4805c2a0 0x7fbd4805e760 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:37.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd60060870 0x7fbd600f6e20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:37.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 --2- 172.21.15.78:0/3870732981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd600f9920 0x7fbd600f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:37.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3870732981 >> 172.21.15.78:0/3870732981 conn(0x7fbd600f0e20 msgr2=0x7fbd600f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:37.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3870732981 shutdown_connections 2023-12-21T03:52:37.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:37.572+0000 7fbd64eaf700 1 -- 172.21.15.78:0/3870732981 wait complete. 2023-12-21T03:52:37.725 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:35.644642+0000 mgr.smithi078.zidbgq (mgr.14162) 439 : cluster [DBG] pgmap v418: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:37.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:35.644642+0000 mgr.smithi078.zidbgq (mgr.14162) 439 : cluster [DBG] pgmap v418: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:37.938 INFO:teuthology.orchestra.run.smithi078.stdout:38654705755 2023-12-21T03:52:37.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2023-12-21T03:52:38.159 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:39.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.165+0000 7f46878b2700 1 -- 172.21.15.78:0/67458383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46800f7a70 msgr2=0x7f46800f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:39.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.165+0000 7f46878b2700 1 --2- 172.21.15.78:0/67458383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46800f7a70 0x7f46800f7ed0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f4670004660 tx=0x7f4670009b30).stop 2023-12-21T03:52:39.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.165+0000 7f46878b2700 1 -- 172.21.15.78:0/67458383 shutdown_connections 2023-12-21T03:52:39.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.165+0000 7f46878b2700 1 --2- 172.21.15.78:0/67458383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46800f7a70 0x7f46800f7ed0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:39.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.165+0000 7f46878b2700 1 --2- 172.21.15.78:0/67458383 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46800f67d0 0x7f46800f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.165+0000 7f46878b2700 1 -- 172.21.15.78:0/67458383 >> 172.21.15.78:0/67458383 conn(0x7f46800f1f50 msgr2=0x7f46800f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.165+0000 7f46878b2700 1 -- 172.21.15.78:0/67458383 shutdown_connections 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.165+0000 7f46878b2700 1 -- 172.21.15.78:0/67458383 wait complete. 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f46878b2700 1 Processor -- start 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f46878b2700 1 -- start start 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f46878b2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46800f67d0 0x7f46801070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f46878b2700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46800f7a70 0x7f46801075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f46878b2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4680107b60 con 0x7f46800f67d0 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f46878b2700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4680107ca0 con 0x7f46800f7a70 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f468564e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46800f67d0 0x7f46801070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f468564e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46800f67d0 0x7f46801070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53858/0 (socket says 172.21.15.78:53858) 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f468564e700 1 -- 172.21.15.78:0/2824549316 learned_addr learned my addr 172.21.15.78:0/2824549316 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f4684e4d700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46800f7a70 0x7f46801075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:39.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f468564e700 1 -- 172.21.15.78:0/2824549316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46800f7a70 msgr2=0x7f46801075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f468564e700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46800f7a70 0x7f46801075b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f468564e700 1 -- 172.21.15.78:0/2824549316 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4670005040 con 0x7f46800f67d0 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f4684e4d700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46800f7a70 0x7f46801075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f468564e700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46800f67d0 0x7f46801070c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f467c002700 tx=0x7f467c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.166+0000 7f4675ffb700 1 -- 172.21.15.78:0/2824549316 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f467c009320 con 0x7f46800f67d0 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.167+0000 7f4675ffb700 1 -- 172.21.15.78:0/2824549316 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f467c00f040 con 0x7f46800f67d0 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.167+0000 7f46878b2700 1 -- 172.21.15.78:0/2824549316 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4680108640 con 0x7f46800f67d0 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.167+0000 7f4675ffb700 1 -- 172.21.15.78:0/2824549316 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f467c014900 con 0x7f46800f67d0 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.167+0000 7f46878b2700 1 -- 172.21.15.78:0/2824549316 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4680108c00 con 0x7f46800f67d0 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.168+0000 7f46878b2700 1 -- 172.21.15.78:0/2824549316 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f468005dd10 con 0x7f46800f67d0 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.168+0000 7f4675ffb700 1 -- 172.21.15.78:0/2824549316 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f467c009480 con 0x7f46800f67d0 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.168+0000 7f4675ffb700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f466c05c2a0 0x7f466c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.168+0000 7f4675ffb700 1 -- 172.21.15.78:0/2824549316 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f467c075ed0 con 0x7f46800f67d0 2023-12-21T03:52:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.168+0000 7f4684e4d700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f466c05c2a0 0x7f466c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:39.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.169+0000 7f4675ffb700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419] conn(0x7f466c062030 0x7f466c064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:39.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.169+0000 7f4675ffb700 1 -- 172.21.15.78:0/2824549316 --> [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f466c064b10 con 0x7f466c062030 2023-12-21T03:52:39.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.169+0000 7f4675ffb700 1 -- 172.21.15.78:0/2824549316 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f467c078540 con 0x7f46800f67d0 2023-12-21T03:52:39.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.169+0000 7f4685e4f700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419] conn(0x7f466c062030 0x7f466c064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:39.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.169+0000 7f4684e4d700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f466c05c2a0 0x7f466c05e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4670005010 tx=0x7f4670009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:39.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.169+0000 7f4685e4f700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419] conn(0x7f466c062030 0x7f466c064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:39.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.170+0000 7f4675ffb700 1 -- 172.21.15.78:0/2824549316 <== osd.1 v2:172.21.15.78:6810/180184419 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f466c064b10 con 0x7f466c062030 2023-12-21T03:52:39.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.183+0000 7f46878b2700 1 -- 172.21.15.78:0/2824549316 --> [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f46801090c0 con 0x7f466c062030 2023-12-21T03:52:39.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f4675ffb700 1 -- 172.21.15.78:0/2824549316 <== osd.1 v2:172.21.15.78:6810/180184419 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f46801090c0 con 0x7f466c062030 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 -- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419] conn(0x7f466c062030 msgr2=0x7f466c064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419] conn(0x7f466c062030 0x7f466c064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 -- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f466c05c2a0 msgr2=0x7f466c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f466c05c2a0 0x7f466c05e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4670005010 tx=0x7f4670009b30).stop 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 -- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46800f67d0 msgr2=0x7f46801070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46800f67d0 0x7f46801070c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f467c002700 tx=0x7f467c009d80).stop 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 -- 172.21.15.78:0/2824549316 shutdown_connections 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6810/180184419,v1:172.21.15.78:6811/180184419] conn(0x7f466c062030 0x7f466c064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f466c05c2a0 0x7f466c05e760 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46800f67d0 0x7f46801070c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 --2- 172.21.15.78:0/2824549316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46800f7a70 0x7f46801075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:39.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 -- 172.21.15.78:0/2824549316 >> 172.21.15.78:0/2824549316 conn(0x7f46800f1f50 msgr2=0x7f46800fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:39.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 -- 172.21.15.78:0/2824549316 shutdown_connections 2023-12-21T03:52:39.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:39.184+0000 7f46878b2700 1 -- 172.21.15.78:0/2824549316 wait complete. 2023-12-21T03:52:39.553 INFO:teuthology.orchestra.run.smithi078.stdout:55834574937 2023-12-21T03:52:39.554 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2023-12-21T03:52:39.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:37.644907+0000 mgr.smithi078.zidbgq (mgr.14162) 440 : cluster [DBG] pgmap v419: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:39.775 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:39.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:37.644907+0000 mgr.smithi078.zidbgq (mgr.14162) 440 : cluster [DBG] pgmap v419: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.787+0000 7f5781341700 1 -- 172.21.15.78:0/408089979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f577c0f67e0 msgr2=0x7f577c0f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.787+0000 7f5781341700 1 --2- 172.21.15.78:0/408089979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f577c0f67e0 0x7f577c0f6c00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f5770004660 tx=0x7f5770009b30).stop 2023-12-21T03:52:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.787+0000 7f5781341700 1 -- 172.21.15.78:0/408089979 shutdown_connections 2023-12-21T03:52:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.787+0000 7f5781341700 1 --2- 172.21.15.78:0/408089979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f577c0f7a80 0x7f577c0f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.787+0000 7f5781341700 1 --2- 172.21.15.78:0/408089979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f577c0f67e0 0x7f577c0f6c00 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.787+0000 7f5781341700 1 -- 172.21.15.78:0/408089979 >> 172.21.15.78:0/408089979 conn(0x7f577c0f1f20 msgr2=0x7f577c0f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.787+0000 7f5781341700 1 -- 172.21.15.78:0/408089979 shutdown_connections 2023-12-21T03:52:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.787+0000 7f5781341700 1 -- 172.21.15.78:0/408089979 wait complete. 2023-12-21T03:52:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.787+0000 7f5781341700 1 Processor -- start 2023-12-21T03:52:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.787+0000 7f5781341700 1 -- start start 2023-12-21T03:52:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f5781341700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f577c0f67e0 0x7f577c107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f5781341700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f577c0f7a80 0x7f577c107570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f5781341700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c107b20 con 0x7f577c0f7a80 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f5781341700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c107c60 con 0x7f577c0f67e0 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f577a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f577c0f7a80 0x7f577c107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f577a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f577c0f7a80 0x7f577c107570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53880/0 (socket says 172.21.15.78:53880) 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f577a7fc700 1 -- 172.21.15.78:0/576983398 learned_addr learned my addr 172.21.15.78:0/576983398 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f577affd700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f577c0f67e0 0x7f577c107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f577a7fc700 1 -- 172.21.15.78:0/576983398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f577c0f67e0 msgr2=0x7f577c107080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f577a7fc700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f577c0f67e0 0x7f577c107080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f577a7fc700 1 -- 172.21.15.78:0/576983398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5770005040 con 0x7f577c0f7a80 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f577a7fc700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f577c0f7a80 0x7f577c107570 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f576c00e7f0 tx=0x7f576c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f57637fe700 1 -- 172.21.15.78:0/576983398 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f576c0092e0 con 0x7f577c0f7a80 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f5781341700 1 -- 172.21.15.78:0/576983398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577c108600 con 0x7f577c0f7a80 2023-12-21T03:52:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f57637fe700 1 -- 172.21.15.78:0/576983398 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f576c00f040 con 0x7f577c0f7a80 2023-12-21T03:52:40.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f57637fe700 1 -- 172.21.15.78:0/576983398 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f576c0146c0 con 0x7f577c0f7a80 2023-12-21T03:52:40.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.788+0000 7f5781341700 1 -- 172.21.15.78:0/576983398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577c108bc0 con 0x7f577c0f7a80 2023-12-21T03:52:40.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.789+0000 7f5781341700 1 -- 172.21.15.78:0/576983398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f577c05dcf0 con 0x7f577c0f7a80 2023-12-21T03:52:40.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.790+0000 7f57637fe700 1 -- 172.21.15.78:0/576983398 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f576c01b030 con 0x7f577c0f7a80 2023-12-21T03:52:40.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.790+0000 7f57637fe700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f576405c2a0 0x7f576405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:40.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.790+0000 7f57637fe700 1 -- 172.21.15.78:0/576983398 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f576c0759e0 con 0x7f577c0f7a80 2023-12-21T03:52:40.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.790+0000 7f577affd700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f576405c2a0 0x7f576405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:40.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.790+0000 7f57637fe700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027] conn(0x7f5764062080 0x7f57640644e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:40.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.790+0000 7f57637fe700 1 -- 172.21.15.78:0/576983398 --> [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5764064b60 con 0x7f5764062080 2023-12-21T03:52:40.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.790+0000 7f57637fe700 1 -- 172.21.15.78:0/576983398 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f576c075de0 con 0x7f577c0f7a80 2023-12-21T03:52:40.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.790+0000 7f577b7fe700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027] conn(0x7f5764062080 0x7f57640644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:40.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.790+0000 7f577affd700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f576405c2a0 0x7f576405e760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f5770005010 tx=0x7f577000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:40.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.790+0000 7f577b7fe700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027] conn(0x7f5764062080 0x7f57640644e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:40.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.791+0000 7f57637fe700 1 -- 172.21.15.78:0/576983398 <== osd.2 v2:172.21.15.78:6818/56885027 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f5764064b60 con 0x7f5764062080 2023-12-21T03:52:40.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.805+0000 7f5781341700 1 -- 172.21.15.78:0/576983398 --> [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f577c109020 con 0x7f5764062080 2023-12-21T03:52:40.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.805+0000 7f57637fe700 1 -- 172.21.15.78:0/576983398 <== osd.2 v2:172.21.15.78:6818/56885027 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f577c109020 con 0x7f5764062080 2023-12-21T03:52:40.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 -- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027] conn(0x7f5764062080 msgr2=0x7f57640644e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:40.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027] conn(0x7f5764062080 0x7f57640644e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:40.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 -- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f576405c2a0 msgr2=0x7f576405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:40.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f576405c2a0 0x7f576405e760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f5770005010 tx=0x7f577000b040).stop 2023-12-21T03:52:40.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 -- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f577c0f7a80 msgr2=0x7f577c107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:40.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f577c0f7a80 0x7f577c107570 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f576c00e7f0 tx=0x7f576c00b040).stop 2023-12-21T03:52:40.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 -- 172.21.15.78:0/576983398 shutdown_connections 2023-12-21T03:52:40.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6818/56885027,v1:172.21.15.78:6819/56885027] conn(0x7f5764062080 0x7f57640644e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:40.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f576405c2a0 0x7f576405e760 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:40.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f577c0f67e0 0x7f577c107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:40.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 --2- 172.21.15.78:0/576983398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f577c0f7a80 0x7f577c107570 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:40.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 -- 172.21.15.78:0/576983398 >> 172.21.15.78:0/576983398 conn(0x7f577c0f1f20 msgr2=0x7f577c0fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:40.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 -- 172.21.15.78:0/576983398 shutdown_connections 2023-12-21T03:52:40.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:40.806+0000 7f5781341700 1 -- 172.21.15.78:0/576983398 wait complete. 2023-12-21T03:52:41.171 INFO:teuthology.orchestra.run.smithi078.stdout:73014444119 2023-12-21T03:52:41.172 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2023-12-21T03:52:41.392 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:41.736 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:39.645236+0000 mgr.smithi078.zidbgq (mgr.14162) 441 : cluster [DBG] pgmap v420: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:41.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:39.645236+0000 mgr.smithi078.zidbgq (mgr.14162) 441 : cluster [DBG] pgmap v420: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.371+0000 7fc6d60ff700 1 -- 172.21.15.78:0/1200190207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d00f7a70 msgr2=0x7fc6d00f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.371+0000 7fc6d60ff700 1 --2- 172.21.15.78:0/1200190207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d00f7a70 0x7fc6d00f7ed0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fc6c0004660 tx=0x7fc6c0009b30).stop 2023-12-21T03:52:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.371+0000 7fc6d60ff700 1 -- 172.21.15.78:0/1200190207 shutdown_connections 2023-12-21T03:52:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.371+0000 7fc6d60ff700 1 --2- 172.21.15.78:0/1200190207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d00f7a70 0x7fc6d00f7ed0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.371+0000 7fc6d60ff700 1 --2- 172.21.15.78:0/1200190207 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6d00f67d0 0x7fc6d00f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:52:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.371+0000 7fc6d60ff700 1 -- 172.21.15.78:0/1200190207 >> 172.21.15.78:0/1200190207 conn(0x7fc6d00f1f50 msgr2=0x7fc6d00f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.371+0000 7fc6d60ff700 1 -- 172.21.15.78:0/1200190207 shutdown_connections 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.371+0000 7fc6d60ff700 1 -- 172.21.15.78:0/1200190207 wait complete. 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.371+0000 7fc6d60ff700 1 Processor -- start 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.371+0000 7fc6d60ff700 1 -- start start 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6d60ff700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6d00f67d0 0x7fc6d006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6d60ff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d00f7a70 0x7fc6d0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6d60ff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d006a410 con 0x7fc6d00f7a70 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6d60ff700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d006a550 con 0x7fc6d00f67d0 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6ceffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d00f7a70 0x7fc6d0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6ceffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d00f7a70 0x7fc6d0069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53898/0 (socket says 172.21.15.78:53898) 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6ceffd700 1 -- 172.21.15.78:0/2626713651 learned_addr learned my addr 172.21.15.78:0/2626713651 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6cf7fe700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6d00f67d0 0x7fc6d006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6ceffd700 1 -- 172.21.15.78:0/2626713651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6d00f67d0 msgr2=0x7fc6d006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6ceffd700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6d00f67d0 0x7fc6d006b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6ceffd700 1 -- 172.21.15.78:0/2626713651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c0005040 con 0x7fc6d00f7a70 2023-12-21T03:52:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6ceffd700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d00f7a70 0x7fc6d0069e20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fc6c0009830 tx=0x7fc6c0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6b7fff700 1 -- 172.21.15.78:0/2626713651 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c0018070 con 0x7fc6d00f7a70 2023-12-21T03:52:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6d60ff700 1 -- 172.21.15.78:0/2626713651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d006a7d0 con 0x7fc6d00f7a70 2023-12-21T03:52:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6b7fff700 1 -- 172.21.15.78:0/2626713651 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6c0013410 con 0x7fc6d00f7a70 2023-12-21T03:52:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6b7fff700 1 -- 172.21.15.78:0/2626713651 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c001c620 con 0x7fc6d00f7a70 2023-12-21T03:52:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.372+0000 7fc6d60ff700 1 -- 172.21.15.78:0/2626713651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d010f0f0 con 0x7fc6d00f7a70 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.373+0000 7fc6d60ff700 1 -- 172.21.15.78:0/2626713651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc6d005dd10 con 0x7fc6d00f7a70 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.374+0000 7fc6b7fff700 1 -- 172.21.15.78:0/2626713651 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc6c001c8e0 con 0x7fc6d00f7a70 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.374+0000 7fc6b7fff700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc6b805c2a0 0x7fc6b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.374+0000 7fc6b7fff700 1 -- 172.21.15.78:0/2626713651 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc6c0077a70 con 0x7fc6d00f7a70 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.374+0000 7fc6cf7fe700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc6b805c2a0 0x7fc6b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.374+0000 7fc6b7fff700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634] conn(0x7fc6b8062030 0x7fc6b8064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.374+0000 7fc6b7fff700 1 -- 172.21.15.78:0/2626713651 --> [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6b8064b10 con 0x7fc6b8062030 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.374+0000 7fc6b7fff700 1 -- 172.21.15.78:0/2626713651 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6c007c220 con 0x7fc6d00f7a70 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.374+0000 7fc6cf7fe700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc6b805c2a0 0x7fc6b805e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fc6d006b730 tx=0x7fc6c4008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.374+0000 7fc6cffff700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634] conn(0x7fc6b8062030 0x7fc6b8064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.375+0000 7fc6cffff700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634] conn(0x7fc6b8062030 0x7fc6b8064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.375+0000 7fc6b7fff700 1 -- 172.21.15.78:0/2626713651 <== osd.3 v2:172.21.15.148:6800/2694312634 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fc6b8064b10 con 0x7fc6b8062030 2023-12-21T03:52:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 -- 172.21.15.78:0/2626713651 --> [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc6d00581d0 con 0x7fc6b8062030 2023-12-21T03:52:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6b7fff700 1 -- 172.21.15.78:0/2626713651 <== osd.3 v2:172.21.15.148:6800/2694312634 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc6d00581d0 con 0x7fc6b8062030 2023-12-21T03:52:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 -- 172.21.15.78:0/2626713651 >> [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634] conn(0x7fc6b8062030 msgr2=0x7fc6b8064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634] conn(0x7fc6b8062030 0x7fc6b8064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 -- 172.21.15.78:0/2626713651 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc6b805c2a0 msgr2=0x7fc6b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc6b805c2a0 0x7fc6b805e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fc6d006b730 tx=0x7fc6c4008040).stop 2023-12-21T03:52:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 -- 172.21.15.78:0/2626713651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d00f7a70 msgr2=0x7fc6d0069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d00f7a70 0x7fc6d0069e20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fc6c0009830 tx=0x7fc6c0009b30).stop 2023-12-21T03:52:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 -- 172.21.15.78:0/2626713651 shutdown_connections 2023-12-21T03:52:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.148:6800/2694312634,v1:172.21.15.148:6801/2694312634] conn(0x7fc6b8062030 0x7fc6b8064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:42.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc6b805c2a0 0x7fc6b805e760 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:42.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6d00f67d0 0x7fc6d006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:42.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 --2- 172.21.15.78:0/2626713651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d00f7a70 0x7fc6d0069e20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:42.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.390+0000 7fc6d60ff700 1 -- 172.21.15.78:0/2626713651 >> 172.21.15.78:0/2626713651 conn(0x7fc6d00f1f50 msgr2=0x7fc6d00fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:42.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.391+0000 7fc6d60ff700 1 -- 172.21.15.78:0/2626713651 shutdown_connections 2023-12-21T03:52:42.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:42.391+0000 7fc6d60ff700 1 -- 172.21.15.78:0/2626713651 wait complete. 2023-12-21T03:52:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:107374182442 2023-12-21T03:52:42.780 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2023-12-21T03:52:42.999 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:43.707 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:41.645520+0000 mgr.smithi078.zidbgq (mgr.14162) 442 : cluster [DBG] pgmap v421: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:41.645520+0000 mgr.smithi078.zidbgq (mgr.14162) 442 : cluster [DBG] pgmap v421: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.871+0000 7f34e3fff700 1 -- 172.21.15.78:0/2014538095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34e40f67b0 msgr2=0x7f34e40f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.871+0000 7f34e3fff700 1 --2- 172.21.15.78:0/2014538095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34e40f67b0 0x7f34e40f6c10 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f34d4004660 tx=0x7f34d4009b30).stop 2023-12-21T03:52:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.871+0000 7f34e3fff700 1 -- 172.21.15.78:0/2014538095 shutdown_connections 2023-12-21T03:52:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.871+0000 7f34e3fff700 1 --2- 172.21.15.78:0/2014538095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34e40f67b0 0x7f34e40f6c10 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.871+0000 7f34e3fff700 1 --2- 172.21.15.78:0/2014538095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34e40f5510 0x7f34e40f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:52:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.871+0000 7f34e3fff700 1 -- 172.21.15.78:0/2014538095 >> 172.21.15.78:0/2014538095 conn(0x7f34e40f0c70 msgr2=0x7f34e40f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.871+0000 7f34e3fff700 1 -- 172.21.15.78:0/2014538095 shutdown_connections 2023-12-21T03:52:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.871+0000 7f34e3fff700 1 -- 172.21.15.78:0/2014538095 wait complete. 2023-12-21T03:52:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.872+0000 7f34e3fff700 1 Processor -- start 2023-12-21T03:52:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.872+0000 7f34e3fff700 1 -- start start 2023-12-21T03:52:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.872+0000 7f34e3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34e40f5510 0x7f34e4109c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.872+0000 7f34e3fff700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34e40f67b0 0x7f34e410c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.872+0000 7f34e3fff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e40609d0 con 0x7f34e40f5510 2023-12-21T03:52:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.872+0000 7f34e3fff700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e4060b10 con 0x7f34e40f67b0 2023-12-21T03:52:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34e2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34e40f5510 0x7f34e4109c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34e2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34e40f5510 0x7f34e4109c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53924/0 (socket says 172.21.15.78:53924) 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34e2ffd700 1 -- 172.21.15.78:0/2318911889 learned_addr learned my addr 172.21.15.78:0/2318911889 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34e27fc700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34e40f67b0 0x7f34e410c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34e2ffd700 1 -- 172.21.15.78:0/2318911889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34e40f67b0 msgr2=0x7f34e410c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34e2ffd700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34e40f67b0 0x7f34e410c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34e2ffd700 1 -- 172.21.15.78:0/2318911889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d4005040 con 0x7f34e40f5510 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34e27fc700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34e40f67b0 0x7f34e410c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34e2ffd700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34e40f5510 0x7f34e4109c20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f34d80093f0 tx=0x7f34d8006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34cb7fe700 1 -- 172.21.15.78:0/2318911889 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34d8009ea0 con 0x7f34e40f5510 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34cb7fe700 1 -- 172.21.15.78:0/2318911889 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34d8013af0 con 0x7f34e40f5510 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34cb7fe700 1 -- 172.21.15.78:0/2318911889 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34d800dad0 con 0x7f34e40f5510 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34e3fff700 1 -- 172.21.15.78:0/2318911889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e410c780 con 0x7f34e40f5510 2023-12-21T03:52:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.873+0000 7f34e3fff700 1 -- 172.21.15.78:0/2318911889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e410ccc0 con 0x7f34e40f5510 2023-12-21T03:52:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.874+0000 7f34e3fff700 1 -- 172.21.15.78:0/2318911889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f34e405db80 con 0x7f34e40f5510 2023-12-21T03:52:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.874+0000 7f34cb7fe700 1 -- 172.21.15.78:0/2318911889 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f34d801a060 con 0x7f34e40f5510 2023-12-21T03:52:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.875+0000 7f34cb7fe700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f34cc05c2b0 0x7f34cc05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.875+0000 7f34cb7fe700 1 -- 172.21.15.78:0/2318911889 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f34d80761d0 con 0x7f34e40f5510 2023-12-21T03:52:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.875+0000 7f34e27fc700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f34cc05c2b0 0x7f34cc05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.875+0000 7f34cb7fe700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721] conn(0x7f34cc062040 0x7f34cc0644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.875+0000 7f34cb7fe700 1 -- 172.21.15.78:0/2318911889 --> [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34cc064b20 con 0x7f34cc062040 2023-12-21T03:52:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.875+0000 7f34cb7fe700 1 -- 172.21.15.78:0/2318911889 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34d8076510 con 0x7f34e40f5510 2023-12-21T03:52:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.875+0000 7f34e27fc700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f34cc05c2b0 0x7f34cc05e770 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f34d4013580 tx=0x7f34d4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.875+0000 7f34e37fe700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721] conn(0x7f34cc062040 0x7f34cc0644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.876+0000 7f34e37fe700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721] conn(0x7f34cc062040 0x7f34cc0644a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.876+0000 7f34cb7fe700 1 -- 172.21.15.78:0/2318911889 <== osd.4 v2:172.21.15.148:6808/4024298721 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f34cc064b20 con 0x7f34cc062040 2023-12-21T03:52:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.890+0000 7f34e3fff700 1 -- 172.21.15.78:0/2318911889 --> [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f34e4058150 con 0x7f34cc062040 2023-12-21T03:52:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34cb7fe700 1 -- 172.21.15.78:0/2318911889 <== osd.4 v2:172.21.15.148:6808/4024298721 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f34e4058150 con 0x7f34cc062040 2023-12-21T03:52:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 -- 172.21.15.78:0/2318911889 >> [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721] conn(0x7f34cc062040 msgr2=0x7f34cc0644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721] conn(0x7f34cc062040 0x7f34cc0644a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 -- 172.21.15.78:0/2318911889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f34cc05c2b0 msgr2=0x7f34cc05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f34cc05c2b0 0x7f34cc05e770 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f34d4013580 tx=0x7f34d4009b30).stop 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 -- 172.21.15.78:0/2318911889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34e40f5510 msgr2=0x7f34e4109c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34e40f5510 0x7f34e4109c20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f34d80093f0 tx=0x7f34d8006ec0).stop 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 -- 172.21.15.78:0/2318911889 shutdown_connections 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.148:6808/4024298721,v1:172.21.15.148:6809/4024298721] conn(0x7f34cc062040 0x7f34cc0644a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f34cc05c2b0 0x7f34cc05e770 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34e40f5510 0x7f34e4109c20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 --2- 172.21.15.78:0/2318911889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34e40f67b0 0x7f34e410c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 -- 172.21.15.78:0/2318911889 >> 172.21.15.78:0/2318911889 conn(0x7f34e40f0c70 msgr2=0x7f34e40fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 -- 172.21.15.78:0/2318911889 shutdown_connections 2023-12-21T03:52:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:43.891+0000 7f34e3fff700 1 -- 172.21.15.78:0/2318911889 wait complete. 2023-12-21T03:52:44.304 INFO:teuthology.orchestra.run.smithi078.stdout:133143986185 2023-12-21T03:52:44.305 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2023-12-21T03:52:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:45.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.437+0000 7f93618ae700 1 -- 172.21.15.78:0/531886703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f935c0f67e0 msgr2=0x7f935c0f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:45.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.437+0000 7f93618ae700 1 --2- 172.21.15.78:0/531886703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f935c0f67e0 0x7f935c0f6c00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f9344004660 tx=0x7f9344009b30).stop 2023-12-21T03:52:45.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.437+0000 7f93618ae700 1 -- 172.21.15.78:0/531886703 shutdown_connections 2023-12-21T03:52:45.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.437+0000 7f93618ae700 1 --2- 172.21.15.78:0/531886703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f935c0f7a80 0x7f935c0f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:45.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.437+0000 7f93618ae700 1 --2- 172.21.15.78:0/531886703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f935c0f67e0 0x7f935c0f6c00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:45.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.437+0000 7f93618ae700 1 -- 172.21.15.78:0/531886703 >> 172.21.15.78:0/531886703 conn(0x7f935c0f1f20 msgr2=0x7f935c0f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:45.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.437+0000 7f93618ae700 1 -- 172.21.15.78:0/531886703 shutdown_connections 2023-12-21T03:52:45.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.437+0000 7f93618ae700 1 -- 172.21.15.78:0/531886703 wait complete. 2023-12-21T03:52:45.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.438+0000 7f93618ae700 1 Processor -- start 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.438+0000 7f93618ae700 1 -- start start 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.438+0000 7f93618ae700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f935c0f67e0 0x7f935c1070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.438+0000 7f93618ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f935c0f7a80 0x7f935c1075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.438+0000 7f93618ae700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c107b70 con 0x7f935c0f7a80 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.438+0000 7f93618ae700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c107cb0 con 0x7f935c0f67e0 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f935a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f935c0f7a80 0x7f935c1075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f935a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f935c0f7a80 0x7f935c1075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53950/0 (socket says 172.21.15.78:53950) 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f935a7fc700 1 -- 172.21.15.78:0/172568140 learned_addr learned my addr 172.21.15.78:0/172568140 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f935affd700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f935c0f67e0 0x7f935c1070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f935a7fc700 1 -- 172.21.15.78:0/172568140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f935c0f67e0 msgr2=0x7f935c1070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f935a7fc700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f935c0f67e0 0x7f935c1070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f935a7fc700 1 -- 172.21.15.78:0/172568140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9344005040 con 0x7f935c0f7a80 2023-12-21T03:52:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f935a7fc700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f935c0f7a80 0x7f935c1075c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f934c00c790 tx=0x7f934c007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:45.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f93537fe700 1 -- 172.21.15.78:0/172568140 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f934c002590 con 0x7f935c0f7a80 2023-12-21T03:52:45.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f93537fe700 1 -- 172.21.15.78:0/172568140 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f934c002ba0 con 0x7f935c0f7a80 2023-12-21T03:52:45.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f93618ae700 1 -- 172.21.15.78:0/172568140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c108650 con 0x7f935c0f7a80 2023-12-21T03:52:45.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f93537fe700 1 -- 172.21.15.78:0/172568140 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f934c013d90 con 0x7f935c0f7a80 2023-12-21T03:52:45.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.439+0000 7f93618ae700 1 -- 172.21.15.78:0/172568140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c108c10 con 0x7f935c0f7a80 2023-12-21T03:52:45.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.440+0000 7f93618ae700 1 -- 172.21.15.78:0/172568140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f935c05dcf0 con 0x7f935c0f7a80 2023-12-21T03:52:45.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.440+0000 7f93537fe700 1 -- 172.21.15.78:0/172568140 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f934c01a060 con 0x7f935c0f7a80 2023-12-21T03:52:45.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.441+0000 7f93537fe700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f934805c2a0 0x7f934805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:45.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.441+0000 7f93537fe700 1 -- 172.21.15.78:0/172568140 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f934c075e00 con 0x7f935c0f7a80 2023-12-21T03:52:45.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.441+0000 7f935affd700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f934805c2a0 0x7f934805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:45.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.441+0000 7f93537fe700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132] conn(0x7f9348062030 0x7f9348064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:45.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.441+0000 7f93537fe700 1 -- 172.21.15.78:0/172568140 --> [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9348064b10 con 0x7f9348062030 2023-12-21T03:52:45.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.441+0000 7f93537fe700 1 -- 172.21.15.78:0/172568140 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f934c07b410 con 0x7f935c0f7a80 2023-12-21T03:52:45.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.441+0000 7f935affd700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f934805c2a0 0x7f934805e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9344004630 tx=0x7f9344009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:45.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.441+0000 7f935b7fe700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132] conn(0x7f9348062030 0x7f9348064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:45.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.442+0000 7f935b7fe700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132] conn(0x7f9348062030 0x7f9348064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:45.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.442+0000 7f93537fe700 1 -- 172.21.15.78:0/172568140 <== osd.5 v2:172.21.15.148:6816/1364724132 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f9348064b10 con 0x7f9348062030 2023-12-21T03:52:45.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.456+0000 7f93618ae700 1 -- 172.21.15.78:0/172568140 --> [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f935c1090c0 con 0x7f9348062030 2023-12-21T03:52:45.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93537fe700 1 -- 172.21.15.78:0/172568140 <== osd.5 v2:172.21.15.148:6816/1364724132 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f935c1090c0 con 0x7f9348062030 2023-12-21T03:52:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 -- 172.21.15.78:0/172568140 >> [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132] conn(0x7f9348062030 msgr2=0x7f9348064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132] conn(0x7f9348062030 0x7f9348064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 -- 172.21.15.78:0/172568140 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f934805c2a0 msgr2=0x7f934805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f934805c2a0 0x7f934805e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9344004630 tx=0x7f9344009b30).stop 2023-12-21T03:52:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 -- 172.21.15.78:0/172568140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f935c0f7a80 msgr2=0x7f935c1075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f935c0f7a80 0x7f935c1075c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f934c00c790 tx=0x7f934c007d70).stop 2023-12-21T03:52:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 -- 172.21.15.78:0/172568140 shutdown_connections 2023-12-21T03:52:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.148:6816/1364724132,v1:172.21.15.148:6817/1364724132] conn(0x7f9348062030 0x7f9348064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f934805c2a0 0x7f934805e760 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f935c0f67e0 0x7f935c1070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:45.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 --2- 172.21.15.78:0/172568140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f935c0f7a80 0x7f935c1075c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:45.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 -- 172.21.15.78:0/172568140 >> 172.21.15.78:0/172568140 conn(0x7f935c0f1f20 msgr2=0x7f935c0fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:45.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 -- 172.21.15.78:0/172568140 shutdown_connections 2023-12-21T03:52:45.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:45.457+0000 7f93618ae700 1 -- 172.21.15.78:0/172568140 wait complete. 2023-12-21T03:52:45.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:43.645839+0000 mgr.smithi078.zidbgq (mgr.14162) 443 : cluster [DBG] pgmap v422: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:43.645839+0000 mgr.smithi078.zidbgq (mgr.14162) 443 : cluster [DBG] pgmap v422: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:45.836 INFO:teuthology.orchestra.run.smithi078.stdout:150323855368 2023-12-21T03:52:45.836 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.0 2023-12-21T03:52:46.055 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.044+0000 7fd3ea12c700 1 -- 172.21.15.78:0/2599899317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3e40f56c0 msgr2=0x7fd3e40f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.044+0000 7fd3ea12c700 1 --2- 172.21.15.78:0/2599899317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3e40f56c0 0x7fd3e40f5ae0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fd3d8004660 tx=0x7fd3d8009b30).stop 2023-12-21T03:52:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.044+0000 7fd3ea12c700 1 -- 172.21.15.78:0/2599899317 shutdown_connections 2023-12-21T03:52:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.044+0000 7fd3ea12c700 1 --2- 172.21.15.78:0/2599899317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e40f6960 0x7fd3e40f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:52:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.044+0000 7fd3ea12c700 1 --2- 172.21.15.78:0/2599899317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3e40f56c0 0x7fd3e40f5ae0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.044+0000 7fd3ea12c700 1 -- 172.21.15.78:0/2599899317 >> 172.21.15.78:0/2599899317 conn(0x7fd3e40f0e20 msgr2=0x7fd3e40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.044+0000 7fd3ea12c700 1 -- 172.21.15.78:0/2599899317 shutdown_connections 2023-12-21T03:52:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.044+0000 7fd3ea12c700 1 -- 172.21.15.78:0/2599899317 wait complete. 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.044+0000 7fd3ea12c700 1 Processor -- start 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3ea12c700 1 -- start start 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3ea12c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3e40f56c0 0x7fd3e41070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3ea12c700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e40f6960 0x7fd3e41075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3ea12c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e4107c10 con 0x7fd3e40f56c0 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3ea12c700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e41084c0 con 0x7fd3e40f6960 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3e37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3e40f56c0 0x7fd3e41070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3e37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3e40f56c0 0x7fd3e41070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53976/0 (socket says 172.21.15.78:53976) 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3e37fe700 1 -- 172.21.15.78:0/505211195 learned_addr learned my addr 172.21.15.78:0/505211195 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3e2ffd700 1 --2- 172.21.15.78:0/505211195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e40f6960 0x7fd3e41075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3e37fe700 1 -- 172.21.15.78:0/505211195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e40f6960 msgr2=0x7fd3e41075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3e37fe700 1 --2- 172.21.15.78:0/505211195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e40f6960 0x7fd3e41075d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3e37fe700 1 -- 172.21.15.78:0/505211195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d8005040 con 0x7fd3e40f56c0 2023-12-21T03:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3e37fe700 1 --2- 172.21.15.78:0/505211195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3e40f56c0 0x7fd3e41070e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fd3d8009f50 tx=0x7fd3d8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3cbfff700 1 -- 172.21.15.78:0/505211195 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3d8018070 con 0x7fd3e40f56c0 2023-12-21T03:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3ea12c700 1 -- 172.21.15.78:0/505211195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e4108740 con 0x7fd3e40f56c0 2023-12-21T03:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3cbfff700 1 -- 172.21.15.78:0/505211195 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3d8013410 con 0x7fd3e40f56c0 2023-12-21T03:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.045+0000 7fd3cbfff700 1 -- 172.21.15.78:0/505211195 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3d801c620 con 0x7fd3e40f56c0 2023-12-21T03:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.046+0000 7fd3ea12c700 1 -- 172.21.15.78:0/505211195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e4108ca0 con 0x7fd3e40f56c0 2023-12-21T03:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.047+0000 7fd3ea12c700 1 -- 172.21.15.78:0/505211195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e4059100 con 0x7fd3e40f56c0 2023-12-21T03:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.047+0000 7fd3cbfff700 1 -- 172.21.15.78:0/505211195 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd3d801c890 con 0x7fd3e40f56c0 2023-12-21T03:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.047+0000 7fd3cbfff700 1 --2- 172.21.15.78:0/505211195 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3cc05c2a0 0x7fd3cc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.047+0000 7fd3e2ffd700 1 --2- 172.21.15.78:0/505211195 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3cc05c2a0 0x7fd3cc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.047+0000 7fd3cbfff700 1 -- 172.21.15.78:0/505211195 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd3d8077a50 con 0x7fd3e40f56c0 2023-12-21T03:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.047+0000 7fd3e2ffd700 1 --2- 172.21.15.78:0/505211195 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3cc05c2a0 0x7fd3cc05e760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fd3d400b790 tx=0x7fd3d4008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:47.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.051+0000 7fd3cbfff700 1 -- 172.21.15.78:0/505211195 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3d804aa50 con 0x7fd3e40f56c0 2023-12-21T03:52:47.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.200+0000 7fd3ea12c700 1 -- 172.21.15.78:0/505211195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd3e40fe050 con 0x7fd3e40f56c0 2023-12-21T03:52:47.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.201+0000 7fd3cbfff700 1 -- 172.21.15.78:0/505211195 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd3d804e070 con 0x7fd3e40f56c0 2023-12-21T03:52:47.202 INFO:teuthology.orchestra.run.smithi078.stdout:38654705757 2023-12-21T03:52:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.202+0000 7fd3ea12c700 1 -- 172.21.15.78:0/505211195 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3cc05c2a0 msgr2=0x7fd3cc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.202+0000 7fd3ea12c700 1 --2- 172.21.15.78:0/505211195 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3cc05c2a0 0x7fd3cc05e760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fd3d400b790 tx=0x7fd3d4008040).stop 2023-12-21T03:52:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.202+0000 7fd3ea12c700 1 -- 172.21.15.78:0/505211195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3e40f56c0 msgr2=0x7fd3e41070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.202+0000 7fd3ea12c700 1 --2- 172.21.15.78:0/505211195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3e40f56c0 0x7fd3e41070e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fd3d8009f50 tx=0x7fd3d8009b30).stop 2023-12-21T03:52:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.203+0000 7fd3ea12c700 1 -- 172.21.15.78:0/505211195 shutdown_connections 2023-12-21T03:52:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.203+0000 7fd3ea12c700 1 --2- 172.21.15.78:0/505211195 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd3cc05c2a0 0x7fd3cc05e760 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.203+0000 7fd3ea12c700 1 --2- 172.21.15.78:0/505211195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3e40f56c0 0x7fd3e41070e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.203+0000 7fd3ea12c700 1 --2- 172.21.15.78:0/505211195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e40f6960 0x7fd3e41075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.203+0000 7fd3ea12c700 1 -- 172.21.15.78:0/505211195 >> 172.21.15.78:0/505211195 conn(0x7fd3e40f0e20 msgr2=0x7fd3e40603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.203+0000 7fd3ea12c700 1 -- 172.21.15.78:0/505211195 shutdown_connections 2023-12-21T03:52:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:47.203+0000 7fd3ea12c700 1 -- 172.21.15.78:0/505211195 wait complete. 2023-12-21T03:52:47.527 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:45.646136+0000 mgr.smithi078.zidbgq (mgr.14162) 444 : cluster [DBG] pgmap v423: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:47.528 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:47.202296+0000 mon.smithi078 (mon.0) 498 : audit [DBG] from='client.? 172.21.15.78:0/505211195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-21T03:52:47.585 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705755 got 38654705757 for osd.0 2023-12-21T03:52:47.586 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.1 2023-12-21T03:52:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:45.646136+0000 mgr.smithi078.zidbgq (mgr.14162) 444 : cluster [DBG] pgmap v423: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:47.202296+0000 mon.smithi078 (mon.0) 498 : audit [DBG] from='client.? 172.21.15.78:0/505211195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-21T03:52:47.805 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:48.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.752+0000 7f20eca3a700 1 -- 172.21.15.78:0/3835344984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20e80f7a60 msgr2=0x7f20e80f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:48.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.752+0000 7f20eca3a700 1 --2- 172.21.15.78:0/3835344984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20e80f7a60 0x7f20e80f7ec0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f20d8004660 tx=0x7f20d8009b30).stop 2023-12-21T03:52:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.752+0000 7f20eca3a700 1 -- 172.21.15.78:0/3835344984 shutdown_connections 2023-12-21T03:52:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.752+0000 7f20eca3a700 1 --2- 172.21.15.78:0/3835344984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20e80f7a60 0x7f20e80f7ec0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.752+0000 7f20eca3a700 1 --2- 172.21.15.78:0/3835344984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20e80f67c0 0x7f20e80f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:52:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.752+0000 7f20eca3a700 1 -- 172.21.15.78:0/3835344984 >> 172.21.15.78:0/3835344984 conn(0x7f20e80f1f20 msgr2=0x7f20e80f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.752+0000 7f20eca3a700 1 -- 172.21.15.78:0/3835344984 shutdown_connections 2023-12-21T03:52:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.752+0000 7f20eca3a700 1 -- 172.21.15.78:0/3835344984 wait complete. 2023-12-21T03:52:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.752+0000 7f20eca3a700 1 Processor -- start 2023-12-21T03:52:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.752+0000 7f20eca3a700 1 -- start start 2023-12-21T03:52:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20eca3a700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20e80f67c0 0x7f20e806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20eca3a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20e80f7a60 0x7f20e8069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20eca3a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e806a410 con 0x7f20e80f7a60 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20eca3a700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e806a550 con 0x7f20e80f67c0 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20e5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20e80f7a60 0x7f20e8069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20e5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20e80f7a60 0x7f20e8069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39220/0 (socket says 172.21.15.78:39220) 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20e5d9b700 1 -- 172.21.15.78:0/107371290 learned_addr learned my addr 172.21.15.78:0/107371290 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20e5d9b700 1 -- 172.21.15.78:0/107371290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20e80f67c0 msgr2=0x7f20e806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20e659c700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20e80f67c0 0x7f20e806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20e5d9b700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20e80f67c0 0x7f20e806b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20e5d9b700 1 -- 172.21.15.78:0/107371290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d8005040 con 0x7f20e80f7a60 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20e659c700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20e80f67c0 0x7f20e806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20e5d9b700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20e80f7a60 0x7f20e8069e20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f20d8009830 tx=0x7f20d8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20d6ffd700 1 -- 172.21.15.78:0/107371290 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20d8018070 con 0x7f20e80f7a60 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20d6ffd700 1 -- 172.21.15.78:0/107371290 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20d8013410 con 0x7f20e80f7a60 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20eca3a700 1 -- 172.21.15.78:0/107371290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e806a7d0 con 0x7f20e80f7a60 2023-12-21T03:52:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20d6ffd700 1 -- 172.21.15.78:0/107371290 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20d801c620 con 0x7f20e80f7a60 2023-12-21T03:52:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.753+0000 7f20eca3a700 1 -- 172.21.15.78:0/107371290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e810f0e0 con 0x7f20e80f7a60 2023-12-21T03:52:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.754+0000 7f20eca3a700 1 -- 172.21.15.78:0/107371290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e8045ad0 con 0x7f20e80f7a60 2023-12-21T03:52:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.755+0000 7f20d6ffd700 1 -- 172.21.15.78:0/107371290 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f20d801c8e0 con 0x7f20e80f7a60 2023-12-21T03:52:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.755+0000 7f20d6ffd700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f20d005c2a0 0x7f20d005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.755+0000 7f20d6ffd700 1 -- 172.21.15.78:0/107371290 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f20d8077dc0 con 0x7f20e80f7a60 2023-12-21T03:52:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.755+0000 7f20e659c700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f20d005c2a0 0x7f20d005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.756+0000 7f20e659c700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f20d005c2a0 0x7f20d005e760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f20e806b730 tx=0x7f20dc008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.759+0000 7f20d6ffd700 1 -- 172.21.15.78:0/107371290 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20d804ae50 con 0x7f20e80f7a60 2023-12-21T03:52:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.907+0000 7f20eca3a700 1 -- 172.21.15.78:0/107371290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f20e8027a00 con 0x7f20e80f7a60 2023-12-21T03:52:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.908+0000 7f20d6ffd700 1 -- 172.21.15.78:0/107371290 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f20d804e470 con 0x7f20e80f7a60 2023-12-21T03:52:48.911 INFO:teuthology.orchestra.run.smithi078.stdout:55834574938 2023-12-21T03:52:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.909+0000 7f20eca3a700 1 -- 172.21.15.78:0/107371290 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f20d005c2a0 msgr2=0x7f20d005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.909+0000 7f20eca3a700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f20d005c2a0 0x7f20d005e760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f20e806b730 tx=0x7f20dc008040).stop 2023-12-21T03:52:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.909+0000 7f20eca3a700 1 -- 172.21.15.78:0/107371290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20e80f7a60 msgr2=0x7f20e8069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.909+0000 7f20eca3a700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20e80f7a60 0x7f20e8069e20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f20d8009830 tx=0x7f20d8009b30).stop 2023-12-21T03:52:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.910+0000 7f20eca3a700 1 -- 172.21.15.78:0/107371290 shutdown_connections 2023-12-21T03:52:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.910+0000 7f20eca3a700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f20d005c2a0 0x7f20d005e760 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.910+0000 7f20eca3a700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20e80f67c0 0x7f20e806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.910+0000 7f20eca3a700 1 --2- 172.21.15.78:0/107371290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20e80f7a60 0x7f20e8069e20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.910+0000 7f20eca3a700 1 -- 172.21.15.78:0/107371290 >> 172.21.15.78:0/107371290 conn(0x7f20e80f1f20 msgr2=0x7f20e80fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.910+0000 7f20eca3a700 1 -- 172.21.15.78:0/107371290 shutdown_connections 2023-12-21T03:52:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:48.910+0000 7f20eca3a700 1 -- 172.21.15.78:0/107371290 wait complete. 2023-12-21T03:52:49.285 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574937 got 55834574938 for osd.1 2023-12-21T03:52:49.285 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.2 2023-12-21T03:52:49.510 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:49.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:47.646444+0000 mgr.smithi078.zidbgq (mgr.14162) 445 : cluster [DBG] pgmap v424: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:49.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:48.909068+0000 mon.smithi078 (mon.0) 499 : audit [DBG] from='client.? 172.21.15.78:0/107371290' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-21T03:52:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:47.646444+0000 mgr.smithi078.zidbgq (mgr.14162) 445 : cluster [DBG] pgmap v424: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:48.909068+0000 mon.smithi078 (mon.0) 499 : audit [DBG] from='client.? 172.21.15.78:0/107371290' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-21T03:52:50.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.467+0000 7f7cf50f4700 1 -- 172.21.15.78:0/2087256191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf00f6960 msgr2=0x7f7cf00f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:50.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.467+0000 7f7cf50f4700 1 --2- 172.21.15.78:0/2087256191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf00f6960 0x7f7cf00f6dc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f7ce4004660 tx=0x7f7ce4009b30).stop 2023-12-21T03:52:50.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.467+0000 7f7cf50f4700 1 -- 172.21.15.78:0/2087256191 shutdown_connections 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.467+0000 7f7cf50f4700 1 --2- 172.21.15.78:0/2087256191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf00f6960 0x7f7cf00f6dc0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.467+0000 7f7cf50f4700 1 --2- 172.21.15.78:0/2087256191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf00f56c0 0x7f7cf00f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.467+0000 7f7cf50f4700 1 -- 172.21.15.78:0/2087256191 >> 172.21.15.78:0/2087256191 conn(0x7f7cf00f0e20 msgr2=0x7f7cf00f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.467+0000 7f7cf50f4700 1 -- 172.21.15.78:0/2087256191 shutdown_connections 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.467+0000 7f7cf50f4700 1 -- 172.21.15.78:0/2087256191 wait complete. 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.468+0000 7f7cf50f4700 1 Processor -- start 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.468+0000 7f7cf50f4700 1 -- start start 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.468+0000 7f7cf50f4700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf00f56c0 0x7f7cf01070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.468+0000 7f7cf50f4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf00f6960 0x7f7cf01075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.468+0000 7f7cf50f4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0107b70 con 0x7f7cf00f6960 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.468+0000 7f7cf50f4700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0107cb0 con 0x7f7cf00f56c0 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.468+0000 7f7cee59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf00f6960 0x7f7cf01075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.468+0000 7f7cee59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf00f6960 0x7f7cf01075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39240/0 (socket says 172.21.15.78:39240) 2023-12-21T03:52:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.468+0000 7f7cee59c700 1 -- 172.21.15.78:0/3122255691 learned_addr learned my addr 172.21.15.78:0/3122255691 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.469+0000 7f7ceed9d700 1 --2- 172.21.15.78:0/3122255691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf00f56c0 0x7f7cf01070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.469+0000 7f7cee59c700 1 -- 172.21.15.78:0/3122255691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf00f56c0 msgr2=0x7f7cf01070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.469+0000 7f7cee59c700 1 --2- 172.21.15.78:0/3122255691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf00f56c0 0x7f7cf01070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.469+0000 7f7cee59c700 1 -- 172.21.15.78:0/3122255691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce4005040 con 0x7f7cf00f6960 2023-12-21T03:52:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.469+0000 7f7cee59c700 1 --2- 172.21.15.78:0/3122255691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf00f6960 0x7f7cf01075c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f7ce4009830 tx=0x7f7ce4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.469+0000 7f7cd77fe700 1 -- 172.21.15.78:0/3122255691 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ce4018070 con 0x7f7cf00f6960 2023-12-21T03:52:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.469+0000 7f7cd77fe700 1 -- 172.21.15.78:0/3122255691 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ce4013410 con 0x7f7cf00f6960 2023-12-21T03:52:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.469+0000 7f7cf50f4700 1 -- 172.21.15.78:0/3122255691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf01085f0 con 0x7f7cf00f6960 2023-12-21T03:52:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.469+0000 7f7cd77fe700 1 -- 172.21.15.78:0/3122255691 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ce401c620 con 0x7f7cf00f6960 2023-12-21T03:52:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.469+0000 7f7cf50f4700 1 -- 172.21.15.78:0/3122255691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf0108b50 con 0x7f7cf00f6960 2023-12-21T03:52:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.470+0000 7f7cf50f4700 1 -- 172.21.15.78:0/3122255691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf0045ad0 con 0x7f7cf00f6960 2023-12-21T03:52:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.470+0000 7f7cd77fe700 1 -- 172.21.15.78:0/3122255691 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7ce401c890 con 0x7f7cf00f6960 2023-12-21T03:52:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.470+0000 7f7cd77fe700 1 --2- 172.21.15.78:0/3122255691 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7cd805c2a0 0x7f7cd805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.471+0000 7f7cd77fe700 1 -- 172.21.15.78:0/3122255691 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7ce4077980 con 0x7f7cf00f6960 2023-12-21T03:52:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.471+0000 7f7ceed9d700 1 --2- 172.21.15.78:0/3122255691 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7cd805c2a0 0x7f7cd805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.471+0000 7f7ceed9d700 1 --2- 172.21.15.78:0/3122255691 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7cd805c2a0 0x7f7cd805e760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f7cf00f67c0 tx=0x7f7ce0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.474+0000 7f7cd77fe700 1 -- 172.21.15.78:0/3122255691 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ce404aa00 con 0x7f7cf00f6960 2023-12-21T03:52:50.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.624+0000 7f7cf50f4700 1 -- 172.21.15.78:0/3122255691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7cf01091e0 con 0x7f7cf00f6960 2023-12-21T03:52:50.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.624+0000 7f7cd77fe700 1 -- 172.21.15.78:0/3122255691 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7ce404e020 con 0x7f7cf00f6960 2023-12-21T03:52:50.626 INFO:teuthology.orchestra.run.smithi078.stdout:73014444121 2023-12-21T03:52:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.626+0000 7f7cf50f4700 1 -- 172.21.15.78:0/3122255691 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7cd805c2a0 msgr2=0x7f7cd805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.626+0000 7f7cf50f4700 1 --2- 172.21.15.78:0/3122255691 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7cd805c2a0 0x7f7cd805e760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f7cf00f67c0 tx=0x7f7ce0009040).stop 2023-12-21T03:52:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.626+0000 7f7cf50f4700 1 -- 172.21.15.78:0/3122255691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf00f6960 msgr2=0x7f7cf01075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.626+0000 7f7cf50f4700 1 --2- 172.21.15.78:0/3122255691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf00f6960 0x7f7cf01075c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f7ce4009830 tx=0x7f7ce4009b30).stop 2023-12-21T03:52:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.626+0000 7f7cf50f4700 1 -- 172.21.15.78:0/3122255691 shutdown_connections 2023-12-21T03:52:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.626+0000 7f7cf50f4700 1 --2- 172.21.15.78:0/3122255691 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7cd805c2a0 0x7f7cd805e760 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.626+0000 7f7cf50f4700 1 --2- 172.21.15.78:0/3122255691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf00f56c0 0x7f7cf01070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.626+0000 7f7cf50f4700 1 --2- 172.21.15.78:0/3122255691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf00f6960 0x7f7cf01075c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:50.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.626+0000 7f7cf50f4700 1 -- 172.21.15.78:0/3122255691 >> 172.21.15.78:0/3122255691 conn(0x7f7cf00f0e20 msgr2=0x7f7cf00603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:50.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.626+0000 7f7cf50f4700 1 -- 172.21.15.78:0/3122255691 shutdown_connections 2023-12-21T03:52:50.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:50.626+0000 7f7cf50f4700 1 -- 172.21.15.78:0/3122255691 wait complete. 2023-12-21T03:52:51.044 INFO:tasks.cephadm.ceph_manager.ceph:need seq 73014444119 got 73014444121 for osd.2 2023-12-21T03:52:51.044 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.3 2023-12-21T03:52:51.264 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:51.740 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:49.646786+0000 mgr.smithi078.zidbgq (mgr.14162) 446 : cluster [DBG] pgmap v425: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:50.625825+0000 mon.smithi078 (mon.0) 500 : audit [DBG] from='client.? 172.21.15.78:0/3122255691' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-21T03:52:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:49.646786+0000 mgr.smithi078.zidbgq (mgr.14162) 446 : cluster [DBG] pgmap v425: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:50.625825+0000 mon.smithi078 (mon.0) 500 : audit [DBG] from='client.? 172.21.15.78:0/3122255691' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-21T03:52:52.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.229+0000 7f7a77b0a700 1 -- 172.21.15.78:0/395901673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a700f67c0 msgr2=0x7f7a700f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:52.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 --2- 172.21.15.78:0/395901673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a700f67c0 0x7f7a700f6be0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f7a6c004660 tx=0x7f7a6c009b30).stop 2023-12-21T03:52:52.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 -- 172.21.15.78:0/395901673 shutdown_connections 2023-12-21T03:52:52.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 --2- 172.21.15.78:0/395901673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a700f7a60 0x7f7a700f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:52:52.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 --2- 172.21.15.78:0/395901673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a700f67c0 0x7f7a700f6be0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:52.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 -- 172.21.15.78:0/395901673 >> 172.21.15.78:0/395901673 conn(0x7f7a700f1f20 msgr2=0x7f7a700f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 -- 172.21.15.78:0/395901673 shutdown_connections 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 -- 172.21.15.78:0/395901673 wait complete. 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 Processor -- start 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 -- start start 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a700f67c0 0x7f7a701070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a700f7a60 0x7f7a701075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a70107c00 con 0x7f7a700f7a60 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a77b0a700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a701084b0 con 0x7f7a700f67c0 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.230+0000 7f7a750a5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a700f7a60 0x7f7a701075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a750a5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a700f7a60 0x7f7a701075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39252/0 (socket says 172.21.15.78:39252) 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a750a5700 1 -- 172.21.15.78:0/3319003650 learned_addr learned my addr 172.21.15.78:0/3319003650 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a758a6700 1 --2- 172.21.15.78:0/3319003650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a700f67c0 0x7f7a701070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a750a5700 1 -- 172.21.15.78:0/3319003650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a700f67c0 msgr2=0x7f7a701070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:52.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a750a5700 1 --2- 172.21.15.78:0/3319003650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a700f67c0 0x7f7a701070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:52.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a750a5700 1 -- 172.21.15.78:0/3319003650 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c005040 con 0x7f7a700f7a60 2023-12-21T03:52:52.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a750a5700 1 --2- 172.21.15.78:0/3319003650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a700f7a60 0x7f7a701075c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f7a6000b410 tx=0x7f7a60006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:52.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a667fc700 1 -- 172.21.15.78:0/3319003650 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a600125f0 con 0x7f7a700f7a60 2023-12-21T03:52:52.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a667fc700 1 -- 172.21.15.78:0/3319003650 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a60012c00 con 0x7f7a700f7a60 2023-12-21T03:52:52.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a667fc700 1 -- 172.21.15.78:0/3319003650 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a60002aa0 con 0x7f7a700f7a60 2023-12-21T03:52:52.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a77b0a700 1 -- 172.21.15.78:0/3319003650 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a70108790 con 0x7f7a700f7a60 2023-12-21T03:52:52.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.231+0000 7f7a77b0a700 1 -- 172.21.15.78:0/3319003650 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a70108d50 con 0x7f7a700f7a60 2023-12-21T03:52:52.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.232+0000 7f7a77b0a700 1 -- 172.21.15.78:0/3319003650 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a70059100 con 0x7f7a700f7a60 2023-12-21T03:52:52.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.232+0000 7f7a667fc700 1 -- 172.21.15.78:0/3319003650 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7a6000f030 con 0x7f7a700f7a60 2023-12-21T03:52:52.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.233+0000 7f7a667fc700 1 --2- 172.21.15.78:0/3319003650 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a5c05c2a0 0x7f7a5c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:52.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.233+0000 7f7a667fc700 1 -- 172.21.15.78:0/3319003650 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7a60076060 con 0x7f7a700f7a60 2023-12-21T03:52:52.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.233+0000 7f7a758a6700 1 --2- 172.21.15.78:0/3319003650 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a5c05c2a0 0x7f7a5c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:52.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.233+0000 7f7a758a6700 1 --2- 172.21.15.78:0/3319003650 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a5c05c2a0 0x7f7a5c05e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f7a6c004e30 tx=0x7f7a6c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:52.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.237+0000 7f7a667fc700 1 -- 172.21.15.78:0/3319003650 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a60049000 con 0x7f7a700f7a60 2023-12-21T03:52:52.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.388+0000 7f7a77b0a700 1 -- 172.21.15.78:0/3319003650 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7a700fefd0 con 0x7f7a700f7a60 2023-12-21T03:52:52.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.389+0000 7f7a667fc700 1 -- 172.21.15.78:0/3319003650 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7a6004c620 con 0x7f7a700f7a60 2023-12-21T03:52:52.390 INFO:teuthology.orchestra.run.smithi078.stdout:107374182443 2023-12-21T03:52:52.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.390+0000 7f7a77b0a700 1 -- 172.21.15.78:0/3319003650 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a5c05c2a0 msgr2=0x7f7a5c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:52.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.390+0000 7f7a77b0a700 1 --2- 172.21.15.78:0/3319003650 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a5c05c2a0 0x7f7a5c05e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f7a6c004e30 tx=0x7f7a6c009b30).stop 2023-12-21T03:52:52.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.390+0000 7f7a77b0a700 1 -- 172.21.15.78:0/3319003650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a700f7a60 msgr2=0x7f7a701075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:52.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.390+0000 7f7a77b0a700 1 --2- 172.21.15.78:0/3319003650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a700f7a60 0x7f7a701075c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f7a6000b410 tx=0x7f7a60006ef0).stop 2023-12-21T03:52:52.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.390+0000 7f7a77b0a700 1 -- 172.21.15.78:0/3319003650 shutdown_connections 2023-12-21T03:52:52.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.390+0000 7f7a77b0a700 1 --2- 172.21.15.78:0/3319003650 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7a5c05c2a0 0x7f7a5c05e760 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:52.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.390+0000 7f7a77b0a700 1 --2- 172.21.15.78:0/3319003650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a700f67c0 0x7f7a701070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:52.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.390+0000 7f7a77b0a700 1 --2- 172.21.15.78:0/3319003650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a700f7a60 0x7f7a701075c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:52.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.390+0000 7f7a77b0a700 1 -- 172.21.15.78:0/3319003650 >> 172.21.15.78:0/3319003650 conn(0x7f7a700f1f20 msgr2=0x7f7a700fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:52.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.390+0000 7f7a77b0a700 1 -- 172.21.15.78:0/3319003650 shutdown_connections 2023-12-21T03:52:52.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:52.390+0000 7f7a77b0a700 1 -- 172.21.15.78:0/3319003650 wait complete. 2023-12-21T03:52:52.676 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:52.390066+0000 mon.smithi078 (mon.0) 501 : audit [DBG] from='client.? 172.21.15.78:0/3319003650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-21T03:52:52.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:52 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:52.390066+0000 mon.smithi078 (mon.0) 501 : audit [DBG] from='client.? 172.21.15.78:0/3319003650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-21T03:52:52.770 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182442 got 107374182443 for osd.3 2023-12-21T03:52:52.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.4 2023-12-21T03:52:52.991 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:53.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:51.647077+0000 mgr.smithi078.zidbgq (mgr.14162) 447 : cluster [DBG] pgmap v426: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:53.806 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:51.647077+0000 mgr.smithi078.zidbgq (mgr.14162) 447 : cluster [DBG] pgmap v426: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:53.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.977+0000 7f1de26a6700 1 -- 172.21.15.78:0/737452069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ddc0f67c0 msgr2=0x7f1ddc0f6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.977+0000 7f1de26a6700 1 --2- 172.21.15.78:0/737452069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ddc0f67c0 0x7f1ddc0f6c20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f1dd8004660 tx=0x7f1dd8009b30).stop 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.977+0000 7f1de26a6700 1 -- 172.21.15.78:0/737452069 shutdown_connections 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.977+0000 7f1de26a6700 1 --2- 172.21.15.78:0/737452069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ddc0f67c0 0x7f1ddc0f6c20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.977+0000 7f1de26a6700 1 --2- 172.21.15.78:0/737452069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ddc0f5520 0x7f1ddc0f5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.977+0000 7f1de26a6700 1 -- 172.21.15.78:0/737452069 >> 172.21.15.78:0/737452069 conn(0x7f1ddc0f0ca0 msgr2=0x7f1ddc0f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.977+0000 7f1de26a6700 1 -- 172.21.15.78:0/737452069 shutdown_connections 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.977+0000 7f1de26a6700 1 -- 172.21.15.78:0/737452069 wait complete. 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.977+0000 7f1de26a6700 1 Processor -- start 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de26a6700 1 -- start start 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de26a6700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ddc0f5520 0x7f1ddc109b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de26a6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ddc0f67c0 0x7f1ddc10c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de26a6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc060940 con 0x7f1ddc0f67c0 2023-12-21T03:52:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de26a6700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc060a80 con 0x7f1ddc0f5520 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de0ea3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ddc0f67c0 0x7f1ddc10c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de0ea3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ddc0f67c0 0x7f1ddc10c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39284/0 (socket says 172.21.15.78:39284) 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de0ea3700 1 -- 172.21.15.78:0/4151779345 learned_addr learned my addr 172.21.15.78:0/4151779345 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de16a4700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ddc0f5520 0x7f1ddc109b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de0ea3700 1 -- 172.21.15.78:0/4151779345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ddc0f5520 msgr2=0x7f1ddc109b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de0ea3700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ddc0f5520 0x7f1ddc109b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de0ea3700 1 -- 172.21.15.78:0/4151779345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd8005040 con 0x7f1ddc0f67c0 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de16a4700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ddc0f5520 0x7f1ddc109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de0ea3700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ddc0f67c0 0x7f1ddc10c030 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f1dd8000c00 tx=0x7f1dd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1dd1ffb700 1 -- 172.21.15.78:0/4151779345 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1dd8019070 con 0x7f1ddc0f67c0 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1dd1ffb700 1 -- 172.21.15.78:0/4151779345 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1dd8007e70 con 0x7f1ddc0f67c0 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1de26a6700 1 -- 172.21.15.78:0/4151779345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ddc10c580 con 0x7f1ddc0f67c0 2023-12-21T03:52:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.978+0000 7f1dd1ffb700 1 -- 172.21.15.78:0/4151779345 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1dd8014d00 con 0x7f1ddc0f67c0 2023-12-21T03:52:53.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.979+0000 7f1de26a6700 1 -- 172.21.15.78:0/4151779345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ddc10cae0 con 0x7f1ddc0f67c0 2023-12-21T03:52:53.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.979+0000 7f1de26a6700 1 -- 172.21.15.78:0/4151779345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ddc045ad0 con 0x7f1ddc0f67c0 2023-12-21T03:52:53.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.980+0000 7f1dd1ffb700 1 -- 172.21.15.78:0/4151779345 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1dd8025070 con 0x7f1ddc0f67c0 2023-12-21T03:52:53.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.980+0000 7f1dd1ffb700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1dc805c2b0 0x7f1dc805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:53.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.980+0000 7f1dd1ffb700 1 -- 172.21.15.78:0/4151779345 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1dd8076bf0 con 0x7f1ddc0f67c0 2023-12-21T03:52:53.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.980+0000 7f1de16a4700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1dc805c2b0 0x7f1dc805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:53.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.980+0000 7f1de16a4700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1dc805c2b0 0x7f1dc805e770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1ddc0605c0 tx=0x7f1dcc00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:53.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:53.984+0000 7f1dd1ffb700 1 -- 172.21.15.78:0/4151779345 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1dd8049b90 con 0x7f1ddc0f67c0 2023-12-21T03:52:54.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.135+0000 7f1de26a6700 1 -- 172.21.15.78:0/4151779345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1ddc0fdd50 con 0x7f1ddc0f67c0 2023-12-21T03:52:54.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.136+0000 7f1dd1ffb700 1 -- 172.21.15.78:0/4151779345 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1dd8023030 con 0x7f1ddc0f67c0 2023-12-21T03:52:54.137 INFO:teuthology.orchestra.run.smithi078.stdout:133143986187 2023-12-21T03:52:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.137+0000 7f1de26a6700 1 -- 172.21.15.78:0/4151779345 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1dc805c2b0 msgr2=0x7f1dc805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.137+0000 7f1de26a6700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1dc805c2b0 0x7f1dc805e770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1ddc0605c0 tx=0x7f1dcc00b040).stop 2023-12-21T03:52:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.137+0000 7f1de26a6700 1 -- 172.21.15.78:0/4151779345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ddc0f67c0 msgr2=0x7f1ddc10c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.137+0000 7f1de26a6700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ddc0f67c0 0x7f1ddc10c030 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f1dd8000c00 tx=0x7f1dd800b040).stop 2023-12-21T03:52:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.137+0000 7f1de26a6700 1 -- 172.21.15.78:0/4151779345 shutdown_connections 2023-12-21T03:52:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.137+0000 7f1de26a6700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1dc805c2b0 0x7f1dc805e770 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.137+0000 7f1de26a6700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ddc0f5520 0x7f1ddc109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.137+0000 7f1de26a6700 1 --2- 172.21.15.78:0/4151779345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ddc0f67c0 0x7f1ddc10c030 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.137+0000 7f1de26a6700 1 -- 172.21.15.78:0/4151779345 >> 172.21.15.78:0/4151779345 conn(0x7f1ddc0f0ca0 msgr2=0x7f1ddc0fa210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.137+0000 7f1de26a6700 1 -- 172.21.15.78:0/4151779345 shutdown_connections 2023-12-21T03:52:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:54.138+0000 7f1de26a6700 1 -- 172.21.15.78:0/4151779345 wait complete. 2023-12-21T03:52:54.510 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986185 got 133143986187 for osd.4 2023-12-21T03:52:54.511 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.5 2023-12-21T03:52:54.731 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:54.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:54 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:54.137222+0000 mon.smithi078 (mon.0) 502 : audit [DBG] from='client.? 172.21.15.78:0/4151779345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-21T03:52:54.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:54 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:54.137222+0000 mon.smithi078 (mon.0) 502 : audit [DBG] from='client.? 172.21.15.78:0/4151779345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-21T03:52:55.523 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:53.647514+0000 mgr.smithi078.zidbgq (mgr.14162) 448 : cluster [DBG] pgmap v427: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:55.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.702+0000 7f199be88700 1 -- 172.21.15.78:0/2239386340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19940f7a80 msgr2=0x7f19940f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.702+0000 7f199be88700 1 --2- 172.21.15.78:0/2239386340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19940f7a80 0x7f19940f7ee0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f1990004660 tx=0x7f1990009b30).stop 2023-12-21T03:52:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.702+0000 7f199be88700 1 -- 172.21.15.78:0/2239386340 shutdown_connections 2023-12-21T03:52:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.702+0000 7f199be88700 1 --2- 172.21.15.78:0/2239386340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19940f7a80 0x7f19940f7ee0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.702+0000 7f199be88700 1 --2- 172.21.15.78:0/2239386340 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19940f67e0 0x7f19940f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.702+0000 7f199be88700 1 -- 172.21.15.78:0/2239386340 >> 172.21.15.78:0/2239386340 conn(0x7f19940f1f20 msgr2=0x7f19940f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.702+0000 7f199be88700 1 -- 172.21.15.78:0/2239386340 shutdown_connections 2023-12-21T03:52:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.702+0000 7f199be88700 1 -- 172.21.15.78:0/2239386340 wait complete. 2023-12-21T03:52:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.702+0000 7f199be88700 1 Processor -- start 2023-12-21T03:52:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.702+0000 7f199be88700 1 -- start start 2023-12-21T03:52:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f199be88700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19940f67e0 0x7f1994107050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f199be88700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19940f7a80 0x7f1994107540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f199be88700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1994107af0 con 0x7f19940f67e0 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f199be88700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1994107c30 con 0x7f19940f7a80 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f1999c24700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19940f67e0 0x7f1994107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f1999c24700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19940f67e0 0x7f1994107050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39302/0 (socket says 172.21.15.78:39302) 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f1999c24700 1 -- 172.21.15.78:0/2955757238 learned_addr learned my addr 172.21.15.78:0/2955757238 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f1999423700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19940f7a80 0x7f1994107540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f1999c24700 1 -- 172.21.15.78:0/2955757238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19940f7a80 msgr2=0x7f1994107540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f1999c24700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19940f7a80 0x7f1994107540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f1999c24700 1 -- 172.21.15.78:0/2955757238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990005040 con 0x7f19940f67e0 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f1999423700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19940f7a80 0x7f1994107540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f1999c24700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19940f67e0 0x7f1994107050 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f198400c530 tx=0x7f1984007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f198a7fc700 1 -- 172.21.15.78:0/2955757238 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f198400f650 con 0x7f19940f67e0 2023-12-21T03:52:55.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f198a7fc700 1 -- 172.21.15.78:0/2955757238 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f198400fc60 con 0x7f19940f67e0 2023-12-21T03:52:55.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f199be88700 1 -- 172.21.15.78:0/2955757238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19941085d0 con 0x7f19940f67e0 2023-12-21T03:52:55.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f198a7fc700 1 -- 172.21.15.78:0/2955757238 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1984012d20 con 0x7f19940f67e0 2023-12-21T03:52:55.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.703+0000 7f199be88700 1 -- 172.21.15.78:0/2955757238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1994108b90 con 0x7f19940f67e0 2023-12-21T03:52:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.704+0000 7f199be88700 1 -- 172.21.15.78:0/2955757238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1994045ad0 con 0x7f19940f67e0 2023-12-21T03:52:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.705+0000 7f198a7fc700 1 -- 172.21.15.78:0/2955757238 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1984019030 con 0x7f19940f67e0 2023-12-21T03:52:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.705+0000 7f198a7fc700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f198005c2a0 0x7f198005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.705+0000 7f198a7fc700 1 -- 172.21.15.78:0/2955757238 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1984075830 con 0x7f19940f67e0 2023-12-21T03:52:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.705+0000 7f1999423700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f198005c2a0 0x7f198005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.705+0000 7f1999423700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f198005c2a0 0x7f198005e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f1990013580 tx=0x7f1990009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:55.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.709+0000 7f198a7fc700 1 -- 172.21.15.78:0/2955757238 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1984048880 con 0x7f19940f67e0 2023-12-21T03:52:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:53.647514+0000 mgr.smithi078.zidbgq (mgr.14162) 448 : cluster [DBG] pgmap v427: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.861+0000 7f199be88700 1 -- 172.21.15.78:0/2955757238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f19940ff0f0 con 0x7f19940f67e0 2023-12-21T03:52:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.861+0000 7f198a7fc700 1 -- 172.21.15.78:0/2955757238 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1984017070 con 0x7f19940f67e0 2023-12-21T03:52:55.863 INFO:teuthology.orchestra.run.smithi078.stdout:150323855369 2023-12-21T03:52:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.863+0000 7f199be88700 1 -- 172.21.15.78:0/2955757238 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f198005c2a0 msgr2=0x7f198005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.863+0000 7f199be88700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f198005c2a0 0x7f198005e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f1990013580 tx=0x7f1990009b30).stop 2023-12-21T03:52:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.863+0000 7f199be88700 1 -- 172.21.15.78:0/2955757238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19940f67e0 msgr2=0x7f1994107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.863+0000 7f199be88700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19940f67e0 0x7f1994107050 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f198400c530 tx=0x7f1984007e10).stop 2023-12-21T03:52:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.863+0000 7f199be88700 1 -- 172.21.15.78:0/2955757238 shutdown_connections 2023-12-21T03:52:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.863+0000 7f199be88700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f198005c2a0 0x7f198005e760 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.863+0000 7f199be88700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19940f67e0 0x7f1994107050 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.863+0000 7f199be88700 1 --2- 172.21.15.78:0/2955757238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19940f7a80 0x7f1994107540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.863+0000 7f199be88700 1 -- 172.21.15.78:0/2955757238 >> 172.21.15.78:0/2955757238 conn(0x7f19940f1f20 msgr2=0x7f19940fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.863+0000 7f199be88700 1 -- 172.21.15.78:0/2955757238 shutdown_connections 2023-12-21T03:52:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:55.863+0000 7f199be88700 1 -- 172.21.15.78:0/2955757238 wait complete. 2023-12-21T03:52:56.194 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855368 got 150323855369 for osd.5 2023-12-21T03:52:56.194 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-12-21T03:52:56.194 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-21T03:52:56.413 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:56.687 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:55.862749+0000 mon.smithi078 (mon.0) 503 : audit [DBG] from='client.? 172.21.15.78:0/2955757238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-21T03:52:56.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:55.862749+0000 mon.smithi078 (mon.0) 503 : audit [DBG] from='client.? 172.21.15.78:0/2955757238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-21T03:52:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.342+0000 7f4bf325c700 1 -- 172.21.15.78:0/2431227970 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bec0f67e0 msgr2=0x7f4bec0f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.342+0000 7f4bf325c700 1 --2- 172.21.15.78:0/2431227970 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bec0f67e0 0x7f4bec0f6c00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f4be0004660 tx=0x7f4be0009b30).stop 2023-12-21T03:52:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.342+0000 7f4bf325c700 1 -- 172.21.15.78:0/2431227970 shutdown_connections 2023-12-21T03:52:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.342+0000 7f4bf325c700 1 --2- 172.21.15.78:0/2431227970 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bec0f7a80 0x7f4bec0f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.342+0000 7f4bf325c700 1 --2- 172.21.15.78:0/2431227970 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bec0f67e0 0x7f4bec0f6c00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.342+0000 7f4bf325c700 1 -- 172.21.15.78:0/2431227970 >> 172.21.15.78:0/2431227970 conn(0x7f4bec0f1f40 msgr2=0x7f4bec0f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.342+0000 7f4bf325c700 1 -- 172.21.15.78:0/2431227970 shutdown_connections 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.342+0000 7f4bf325c700 1 -- 172.21.15.78:0/2431227970 wait complete. 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bf325c700 1 Processor -- start 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bf325c700 1 -- start start 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bf325c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bec0f67e0 0x7f4bec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bf325c700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bec0f7a80 0x7f4bec069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bf325c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec06a380 con 0x7f4bec0f67e0 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bf325c700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec06a4c0 con 0x7f4bec0f7a80 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bf0ff8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bec0f67e0 0x7f4bec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bf0ff8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bec0f67e0 0x7f4bec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39324/0 (socket says 172.21.15.78:39324) 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bf0ff8700 1 -- 172.21.15.78:0/257589736 learned_addr learned my addr 172.21.15.78:0/257589736 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bebfff700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bec0f7a80 0x7f4bec069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bf0ff8700 1 -- 172.21.15.78:0/257589736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bec0f7a80 msgr2=0x7f4bec069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:57.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.343+0000 7f4bf0ff8700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bec0f7a80 0x7f4bec069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:57.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.344+0000 7f4bf0ff8700 1 -- 172.21.15.78:0/257589736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be0005040 con 0x7f4bec0f67e0 2023-12-21T03:52:57.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.344+0000 7f4bebfff700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bec0f7a80 0x7f4bec069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:52:57.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.344+0000 7f4bf0ff8700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bec0f67e0 0x7f4bec06b9a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4be0009f50 tx=0x7f4be0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:57.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.344+0000 7f4be97fa700 1 -- 172.21.15.78:0/257589736 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4be0018070 con 0x7f4bec0f67e0 2023-12-21T03:52:57.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.344+0000 7f4be97fa700 1 -- 172.21.15.78:0/257589736 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4be0013410 con 0x7f4bec0f67e0 2023-12-21T03:52:57.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.344+0000 7f4bf325c700 1 -- 172.21.15.78:0/257589736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bec06a740 con 0x7f4bec0f67e0 2023-12-21T03:52:57.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.344+0000 7f4be97fa700 1 -- 172.21.15.78:0/257589736 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4be001c620 con 0x7f4bec0f67e0 2023-12-21T03:52:57.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.344+0000 7f4bf325c700 1 -- 172.21.15.78:0/257589736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bec06aca0 con 0x7f4bec0f67e0 2023-12-21T03:52:57.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.345+0000 7f4bf325c700 1 -- 172.21.15.78:0/257589736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bec045ad0 con 0x7f4bec0f67e0 2023-12-21T03:52:57.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.345+0000 7f4be97fa700 1 -- 172.21.15.78:0/257589736 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4be001c780 con 0x7f4bec0f67e0 2023-12-21T03:52:57.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.345+0000 7f4be97fa700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4bdc05c2a0 0x7f4bdc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:57.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.345+0000 7f4be97fa700 1 -- 172.21.15.78:0/257589736 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4be0077950 con 0x7f4bec0f67e0 2023-12-21T03:52:57.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.345+0000 7f4bebfff700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4bdc05c2a0 0x7f4bdc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:57.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.346+0000 7f4bebfff700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4bdc05c2a0 0x7f4bdc05e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4bd80041d0 tx=0x7f4bd8009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.349+0000 7f4be97fa700 1 -- 172.21.15.78:0/257589736 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4be004a9d0 con 0x7f4bec0f67e0 2023-12-21T03:52:57.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.496+0000 7f4bf325c700 1 -- 172.21.15.78:0/257589736 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4bec0ff0e0 con 0x7f4bdc05c2a0 2023-12-21T03:52:57.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.497+0000 7f4be97fa700 1 -- 172.21.15.78:0/257589736 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24027 (secure 0 0 0) 0x7f4bec0ff0e0 con 0x7f4bdc05c2a0 2023-12-21T03:52:57.499 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:52:57.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.499+0000 7f4bf325c700 1 -- 172.21.15.78:0/257589736 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4bdc05c2a0 msgr2=0x7f4bdc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:57.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.499+0000 7f4bf325c700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4bdc05c2a0 0x7f4bdc05e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4bd80041d0 tx=0x7f4bd8009040).stop 2023-12-21T03:52:57.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.499+0000 7f4bf325c700 1 -- 172.21.15.78:0/257589736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bec0f67e0 msgr2=0x7f4bec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:57.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.499+0000 7f4bf325c700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bec0f67e0 0x7f4bec06b9a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4be0009f50 tx=0x7f4be0009b30).stop 2023-12-21T03:52:57.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.499+0000 7f4bf325c700 1 -- 172.21.15.78:0/257589736 shutdown_connections 2023-12-21T03:52:57.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.499+0000 7f4bf325c700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4bdc05c2a0 0x7f4bdc05e760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:57.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.499+0000 7f4bf325c700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bec0f67e0 0x7f4bec06b9a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:57.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.499+0000 7f4bf325c700 1 --2- 172.21.15.78:0/257589736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bec0f7a80 0x7f4bec069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:57.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.499+0000 7f4bf325c700 1 -- 172.21.15.78:0/257589736 >> 172.21.15.78:0/257589736 conn(0x7f4bec0f1f40 msgr2=0x7f4bec0fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:57.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.499+0000 7f4bf325c700 1 -- 172.21.15.78:0/257589736 shutdown_connections 2023-12-21T03:52:57.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:57.499+0000 7f4bf325c700 1 -- 172.21.15.78:0/257589736 wait complete. 2023-12-21T03:52:57.501 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-12-21T03:52:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:55.647859+0000 mgr.smithi078.zidbgq (mgr.14162) 449 : cluster [DBG] pgmap v428: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:57.790 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:55.647859+0000 mgr.smithi078.zidbgq (mgr.14162) 449 : cluster [DBG] pgmap v428: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:57.869 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2023-12-21T03:52:55.647710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31088,"kb_used_data":1920,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":562439824,"statfs":{"total":575970213888,"available":575938379776,"internally_reserved":0,"allocated":1966080,"data_stored":735540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"34","reported_epoch":"35","state":"active+clean","last_fresh":"2023-12-21T03:52:23.501657+0000","last_change":"2023-12-21T03:52:08.770043+0000","last_active":"2023-12-21T03:52:23.501657+0000","last_peered":"2023-12-21T03:52:23.501657+0000","last_clean":"2023-12-21T03:52:23.501657+0000","last_became_active":"2023-12-21T03:52:08.769696+0000","last_became_peered":"2023-12-21T03:52:08.769696+0000","last_unstale":"2023-12-21T03:52:23.501657+0000","last_undegraded":"2023-12-21T03:52:23.501657+0000","last_fullsized":"2023-12-21T03:52:23.501657+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-21T03:49:25.408836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-21T03:49:25.408836+0000","last_clean_scrub_stamp":"2023-12-21T03:49:25.408836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":35,"seq":150323855370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5128,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740024,"statfs":{"total":95995035648,"available":95989784576,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5192,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739960,"statfs":{"total":95995035648,"available":95989719040,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 21 03:52:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60599999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.753,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.33800000000000002}]},{"osd":1,"last update":"Thu Dec 21 03:52:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.54700000000000004,"15min":0.54600000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.73399999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.377},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.74199999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46100000000000002}]},{"osd":2,"last update":"Thu Dec 21 03:52:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76200000000000001,"5min":0.871,"15min":0.871},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69699999999999995,"5min":0.75,"15min":0.75},"last":0.40600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]},{"osd":2,"up_from":17,"seq":73014444122,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5192,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739960,"statfs":{"total":95995035648,"available":95989719040,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 21 03:52:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.34100000000000003,"15min":0.33900000000000002},"min":{"1min":0.224,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.34999999999999998,"15min":0.35099999999999998},"min":{"1min":0.23699999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.23699999999999999}]},{"osd":1,"last update":"Thu Dec 21 03:52:50 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35599999999999998,"15min":0.34599999999999997},"min":{"1min":0.23400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.254,"5min":0.24099999999999999,"15min":0.23599999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.254}]},{"osd":3,"last update":"Thu Dec 21 03:52:40 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.33500000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52900000000000003,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.59899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5256,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739896,"statfs":{"total":95995035648,"available":95989653504,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 21 03:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32900000000000001,"15min":0.34000000000000002},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.496,"5min":0.496,"15min":0.5},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33800000000000002,"15min":0.35699999999999998},"min":{"1min":0.249,"5min":0.247,"15min":0.247},"max":{"1min":0.443,"5min":0.48999999999999999,"15min":0.53800000000000003},"last":0.317}]},{"osd":2,"last update":"Thu Dec 21 03:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35999999999999999,"15min":0.373},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.48799999999999999,"5min":0.496,"15min":0.499},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.35099999999999998,"15min":0.36199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.47399999999999998,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.30099999999999999}]},{"osd":3,"last update":"Thu Dec 21 03:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.52000000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.68799999999999994,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5256,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739896,"statfs":{"total":95995035648,"available":95989653504,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 21 03:52:48 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.33800000000000002,"15min":0.34100000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.46600000000000003,"5min":0.48099999999999998,"15min":0.54600000000000004},"last":0.434},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.35299999999999998,"15min":0.33700000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.19500000000000001},"max":{"1min":0.48399999999999999,"5min":0.53300000000000003,"15min":0.53500000000000003},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Dec 21 03:51:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36199999999999999,"15min":0.34899999999999998},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.23899999999999999},"max":{"1min":0.48799999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36899999999999999,"15min":0.37},"min":{"1min":0.28299999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.65300000000000002},"last":0.41699999999999998}]},{"osd":3,"last update":"Thu Dec 21 03:52:36 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.498,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69399999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-21T03:52:57.870 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-21T03:52:58.090 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:52:58.523 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:58 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:57.497699+0000 mgr.smithi078.zidbgq (mgr.14162) 450 : audit [DBG] from='client.14574 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-21T03:52:58.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:58 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:57.497699+0000 mgr.smithi078.zidbgq (mgr.14162) 450 : audit [DBG] from='client.14574 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-21T03:52:59.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.068+0000 7f09643ce700 1 -- 172.21.15.78:0/525659088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095c060810 msgr2=0x7f095c0f9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:59.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.068+0000 7f09643ce700 1 --2- 172.21.15.78:0/525659088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095c060810 0x7f095c0f9400 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f094c004660 tx=0x7f094c009b30).stop 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.068+0000 7f09643ce700 1 -- 172.21.15.78:0/525659088 shutdown_connections 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.068+0000 7f09643ce700 1 --2- 172.21.15.78:0/525659088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f095c0f9980 0x7f095c0fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.068+0000 7f09643ce700 1 --2- 172.21.15.78:0/525659088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095c060810 0x7f095c0f9400 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.068+0000 7f09643ce700 1 -- 172.21.15.78:0/525659088 >> 172.21.15.78:0/525659088 conn(0x7f095c0f0e20 msgr2=0x7f095c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.068+0000 7f09643ce700 1 -- 172.21.15.78:0/525659088 shutdown_connections 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.068+0000 7f09643ce700 1 -- 172.21.15.78:0/525659088 wait complete. 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f09643ce700 1 Processor -- start 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f09643ce700 1 -- start start 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f09643ce700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f095c060810 0x7f095c107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f09643ce700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095c0f9980 0x7f095c107570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f09643ce700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c107b20 con 0x7f095c0f9980 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f09643ce700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c107c60 con 0x7f095c060810 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f0961969700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095c0f9980 0x7f095c107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:59.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f0961969700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095c0f9980 0x7f095c107570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57378/0 (socket says 172.21.15.78:57378) 2023-12-21T03:52:59.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f0961969700 1 -- 172.21.15.78:0/41565387 learned_addr learned my addr 172.21.15.78:0/41565387 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:52:59.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f096216a700 1 --2- 172.21.15.78:0/41565387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f095c060810 0x7f095c107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:59.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f0961969700 1 -- 172.21.15.78:0/41565387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f095c060810 msgr2=0x7f095c107080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:59.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f0961969700 1 --2- 172.21.15.78:0/41565387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f095c060810 0x7f095c107080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:59.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f0961969700 1 -- 172.21.15.78:0/41565387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094c005040 con 0x7f095c0f9980 2023-12-21T03:52:59.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f0961969700 1 --2- 172.21.15.78:0/41565387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095c0f9980 0x7f095c107570 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f09580026d0 tx=0x7f0958009dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:59.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.069+0000 7f0952ffd700 1 -- 172.21.15.78:0/41565387 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09580079c0 con 0x7f095c0f9980 2023-12-21T03:52:59.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.070+0000 7f0952ffd700 1 -- 172.21.15.78:0/41565387 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f095800f040 con 0x7f095c0f9980 2023-12-21T03:52:59.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.070+0000 7f09643ce700 1 -- 172.21.15.78:0/41565387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c108600 con 0x7f095c0f9980 2023-12-21T03:52:59.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.070+0000 7f0952ffd700 1 -- 172.21.15.78:0/41565387 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0958014840 con 0x7f095c0f9980 2023-12-21T03:52:59.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.070+0000 7f09643ce700 1 -- 172.21.15.78:0/41565387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c108bc0 con 0x7f095c0f9980 2023-12-21T03:52:59.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.071+0000 7f09643ce700 1 -- 172.21.15.78:0/41565387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095c045ad0 con 0x7f095c0f9980 2023-12-21T03:52:59.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.071+0000 7f0952ffd700 1 -- 172.21.15.78:0/41565387 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f095801b060 con 0x7f095c0f9980 2023-12-21T03:52:59.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.071+0000 7f0952ffd700 1 --2- 172.21.15.78:0/41565387 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f094805c2a0 0x7f094805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:52:59.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.071+0000 7f0952ffd700 1 -- 172.21.15.78:0/41565387 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0958075a30 con 0x7f095c0f9980 2023-12-21T03:52:59.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.071+0000 7f096216a700 1 --2- 172.21.15.78:0/41565387 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f094805c2a0 0x7f094805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:52:59.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.072+0000 7f096216a700 1 --2- 172.21.15.78:0/41565387 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f094805c2a0 0x7f094805e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f094c009f50 tx=0x7f094c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:52:59.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.075+0000 7f0952ffd700 1 -- 172.21.15.78:0/41565387 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0958048a30 con 0x7f095c0f9980 2023-12-21T03:52:59.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.221+0000 7f09643ce700 1 -- 172.21.15.78:0/41565387 --> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f095c108e40 con 0x7f094805c2a0 2023-12-21T03:52:59.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.223+0000 7f0952ffd700 1 -- 172.21.15.78:0/41565387 <== mgr.14162 v2:172.21.15.78:6800/2487427889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24027 (secure 0 0 0) 0x7f095c108e40 con 0x7f094805c2a0 2023-12-21T03:52:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:52:59.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.224+0000 7f09643ce700 1 -- 172.21.15.78:0/41565387 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f094805c2a0 msgr2=0x7f094805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:59.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.224+0000 7f09643ce700 1 --2- 172.21.15.78:0/41565387 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f094805c2a0 0x7f094805e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f094c009f50 tx=0x7f094c009b30).stop 2023-12-21T03:52:59.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.224+0000 7f09643ce700 1 -- 172.21.15.78:0/41565387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095c0f9980 msgr2=0x7f095c107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:52:59.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.224+0000 7f09643ce700 1 --2- 172.21.15.78:0/41565387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095c0f9980 0x7f095c107570 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f09580026d0 tx=0x7f0958009dc0).stop 2023-12-21T03:52:59.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.224+0000 7f09643ce700 1 -- 172.21.15.78:0/41565387 shutdown_connections 2023-12-21T03:52:59.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.224+0000 7f09643ce700 1 --2- 172.21.15.78:0/41565387 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f094805c2a0 0x7f094805e760 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:59.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.224+0000 7f09643ce700 1 --2- 172.21.15.78:0/41565387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f095c060810 0x7f095c107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:59.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.224+0000 7f09643ce700 1 --2- 172.21.15.78:0/41565387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095c0f9980 0x7f095c107570 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:52:59.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.224+0000 7f09643ce700 1 -- 172.21.15.78:0/41565387 >> 172.21.15.78:0/41565387 conn(0x7f095c0f0e20 msgr2=0x7f095c0f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:52:59.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.224+0000 7f09643ce700 1 -- 172.21.15.78:0/41565387 shutdown_connections 2023-12-21T03:52:59.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:52:59.224+0000 7f09643ce700 1 -- 172.21.15.78:0/41565387 wait complete. 2023-12-21T03:52:59.231 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-12-21T03:52:59.509 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:52:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:57.648194+0000 mgr.smithi078.zidbgq (mgr.14162) 451 : cluster [DBG] pgmap v429: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:59.610 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2023-12-21T03:52:57.648049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31088,"kb_used_data":1920,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":562439824,"statfs":{"total":575970213888,"available":575938379776,"internally_reserved":0,"allocated":1966080,"data_stored":735540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"34","reported_epoch":"35","state":"active+clean","last_fresh":"2023-12-21T03:52:23.501657+0000","last_change":"2023-12-21T03:52:08.770043+0000","last_active":"2023-12-21T03:52:23.501657+0000","last_peered":"2023-12-21T03:52:23.501657+0000","last_clean":"2023-12-21T03:52:23.501657+0000","last_became_active":"2023-12-21T03:52:08.769696+0000","last_became_peered":"2023-12-21T03:52:08.769696+0000","last_unstale":"2023-12-21T03:52:23.501657+0000","last_undegraded":"2023-12-21T03:52:23.501657+0000","last_fullsized":"2023-12-21T03:52:23.501657+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-21T03:49:25.408836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-21T03:49:25.408836+0000","last_clean_scrub_stamp":"2023-12-21T03:49:25.408836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":35,"seq":150323855370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5128,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740024,"statfs":{"total":95995035648,"available":95989784576,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5192,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739960,"statfs":{"total":95995035648,"available":95989719040,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 21 03:52:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60599999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.753,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.33800000000000002}]},{"osd":1,"last update":"Thu Dec 21 03:52:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.54700000000000004,"15min":0.54600000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.73399999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.377},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.74199999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46100000000000002}]},{"osd":2,"last update":"Thu Dec 21 03:52:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76200000000000001,"5min":0.871,"15min":0.871},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69699999999999995,"5min":0.75,"15min":0.75},"last":0.40600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]},{"osd":2,"up_from":17,"seq":73014444123,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5192,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739960,"statfs":{"total":95995035648,"available":95989719040,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 21 03:52:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.34100000000000003,"15min":0.33900000000000002},"min":{"1min":0.224,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.249},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.34999999999999998,"15min":0.35099999999999998},"min":{"1min":0.23699999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.28499999999999998}]},{"osd":1,"last update":"Thu Dec 21 03:52:50 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35599999999999998,"15min":0.34599999999999997},"min":{"1min":0.23400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.254,"5min":0.24099999999999999,"15min":0.23599999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.29999999999999999}]},{"osd":3,"last update":"Thu Dec 21 03:52:40 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.33500000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52900000000000003,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5256,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739896,"statfs":{"total":95995035648,"available":95989653504,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 21 03:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32900000000000001,"15min":0.34000000000000002},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.496,"5min":0.496,"15min":0.5},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33800000000000002,"15min":0.35699999999999998},"min":{"1min":0.249,"5min":0.247,"15min":0.247},"max":{"1min":0.443,"5min":0.48999999999999999,"15min":0.53800000000000003},"last":0.317}]},{"osd":2,"last update":"Thu Dec 21 03:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35999999999999999,"15min":0.373},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.48799999999999999,"5min":0.496,"15min":0.499},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.35099999999999998,"15min":0.36199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.47399999999999998,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.30099999999999999}]},{"osd":3,"last update":"Thu Dec 21 03:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.52000000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.68799999999999994,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5256,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739896,"statfs":{"total":95995035648,"available":95989653504,"internally_reserved":0,"allocated":327680,"data_stored":122590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 21 03:52:48 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.33800000000000002,"15min":0.34100000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.46600000000000003,"5min":0.48099999999999998,"15min":0.54600000000000004},"last":0.434},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.35299999999999998,"15min":0.33700000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.19500000000000001},"max":{"1min":0.48399999999999999,"5min":0.53300000000000003,"15min":0.53500000000000003},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Dec 21 03:51:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36199999999999999,"15min":0.34899999999999998},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.23899999999999999},"max":{"1min":0.48799999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36899999999999999,"15min":0.37},"min":{"1min":0.28299999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.65300000000000002},"last":0.41699999999999998}]},{"osd":3,"last update":"Thu Dec 21 03:52:36 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.498,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69399999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-21T03:52:59.611 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-12-21T03:52:59.611 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-21T03:52:59.611 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-12-21T03:52:59.611 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:52:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:52:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:57.648194+0000 mgr.smithi078.zidbgq (mgr.14162) 451 : cluster [DBG] pgmap v429: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:52:59.830 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:00.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:52:59.223129+0000 mgr.smithi078.zidbgq (mgr.14162) 452 : audit [DBG] from='client.14578 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.748+0000 7f1bce3b3700 1 -- 172.21.15.78:0/3845675911 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bc80f7a70 msgr2=0x7f1bc80f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.748+0000 7f1bce3b3700 1 --2- 172.21.15.78:0/3845675911 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bc80f7a70 0x7f1bc80f7ed0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f1bb0004660 tx=0x7f1bb0009b30).stop 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 -- 172.21.15.78:0/3845675911 shutdown_connections 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 --2- 172.21.15.78:0/3845675911 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bc80f7a70 0x7f1bc80f7ed0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 --2- 172.21.15.78:0/3845675911 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1bc80f67d0 0x7f1bc80f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 -- 172.21.15.78:0/3845675911 >> 172.21.15.78:0/3845675911 conn(0x7f1bc80f1f50 msgr2=0x7f1bc80f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 -- 172.21.15.78:0/3845675911 shutdown_connections 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 -- 172.21.15.78:0/3845675911 wait complete. 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 Processor -- start 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 -- start start 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bc80f67d0 0x7f1bc81070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1bc80f7a70 0x7f1bc81075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc8107bf0 con 0x7f1bc80f67d0 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.749+0000 7f1bce3b3700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc81084a0 con 0x7f1bc80f7a70 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bc7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bc80f67d0 0x7f1bc81070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bc7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bc80f67d0 0x7f1bc81070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57386/0 (socket says 172.21.15.78:57386) 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bc7fff700 1 -- 172.21.15.78:0/596189319 learned_addr learned my addr 172.21.15.78:0/596189319 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bc77fe700 1 --2- 172.21.15.78:0/596189319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1bc80f7a70 0x7f1bc81075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bc7fff700 1 -- 172.21.15.78:0/596189319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1bc80f7a70 msgr2=0x7f1bc81075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bc7fff700 1 --2- 172.21.15.78:0/596189319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1bc80f7a70 0x7f1bc81075b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bc7fff700 1 -- 172.21.15.78:0/596189319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb0005040 con 0x7f1bc80f67d0 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bc7fff700 1 --2- 172.21.15.78:0/596189319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bc80f67d0 0x7f1bc81070c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f1bb80093f0 tx=0x7f1bb8006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bc4ff9700 1 -- 172.21.15.78:0/596189319 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bb8009ea0 con 0x7f1bc80f67d0 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bc4ff9700 1 -- 172.21.15.78:0/596189319 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bb8013af0 con 0x7f1bc80f67d0 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bce3b3700 1 -- 172.21.15.78:0/596189319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc8108780 con 0x7f1bc80f67d0 2023-12-21T03:53:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bc4ff9700 1 -- 172.21.15.78:0/596189319 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bb800dad0 con 0x7f1bc80f67d0 2023-12-21T03:53:00.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.750+0000 7f1bce3b3700 1 -- 172.21.15.78:0/596189319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc8108d40 con 0x7f1bc80f67d0 2023-12-21T03:53:00.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.751+0000 7f1bce3b3700 1 -- 172.21.15.78:0/596189319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc8059100 con 0x7f1bc80f67d0 2023-12-21T03:53:00.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.751+0000 7f1bc4ff9700 1 -- 172.21.15.78:0/596189319 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1bb801a060 con 0x7f1bc80f67d0 2023-12-21T03:53:00.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.752+0000 7f1bc4ff9700 1 --2- 172.21.15.78:0/596189319 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1bb405c2a0 0x7f1bb405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:00.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.752+0000 7f1bc4ff9700 1 -- 172.21.15.78:0/596189319 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1bb8075ba0 con 0x7f1bc80f67d0 2023-12-21T03:53:00.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.752+0000 7f1bc77fe700 1 --2- 172.21.15.78:0/596189319 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1bb405c2a0 0x7f1bb405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:00.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.752+0000 7f1bc77fe700 1 --2- 172.21.15.78:0/596189319 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1bb405c2a0 0x7f1bb405e760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f1bb0009d60 tx=0x7f1bb0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:00.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.756+0000 7f1bc4ff9700 1 -- 172.21.15.78:0/596189319 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bb8048b40 con 0x7f1bc80f67d0 2023-12-21T03:53:00.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:52:59.223129+0000 mgr.smithi078.zidbgq (mgr.14162) 452 : audit [DBG] from='client.14578 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-21T03:53:00.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.959+0000 7f1bce3b3700 1 -- 172.21.15.78:0/596189319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1bc8045ad0 con 0x7f1bc80f67d0 2023-12-21T03:53:00.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.959+0000 7f1bc4ff9700 1 -- 172.21.15.78:0/596189319 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f1bb804c160 con 0x7f1bc80f67d0 2023-12-21T03:53:00.961 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:00.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.961+0000 7f1bce3b3700 1 -- 172.21.15.78:0/596189319 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1bb405c2a0 msgr2=0x7f1bb405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:00.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.961+0000 7f1bce3b3700 1 --2- 172.21.15.78:0/596189319 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1bb405c2a0 0x7f1bb405e760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f1bb0009d60 tx=0x7f1bb0009b30).stop 2023-12-21T03:53:00.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.961+0000 7f1bce3b3700 1 -- 172.21.15.78:0/596189319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bc80f67d0 msgr2=0x7f1bc81070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:00.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.961+0000 7f1bce3b3700 1 --2- 172.21.15.78:0/596189319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bc80f67d0 0x7f1bc81070c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f1bb80093f0 tx=0x7f1bb8006ec0).stop 2023-12-21T03:53:00.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.961+0000 7f1bce3b3700 1 -- 172.21.15.78:0/596189319 shutdown_connections 2023-12-21T03:53:00.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.961+0000 7f1bce3b3700 1 --2- 172.21.15.78:0/596189319 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1bb405c2a0 0x7f1bb405e760 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:00.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.961+0000 7f1bce3b3700 1 --2- 172.21.15.78:0/596189319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bc80f67d0 0x7f1bc81070c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:00.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.961+0000 7f1bce3b3700 1 --2- 172.21.15.78:0/596189319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1bc80f7a70 0x7f1bc81075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:00.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.961+0000 7f1bce3b3700 1 -- 172.21.15.78:0/596189319 >> 172.21.15.78:0/596189319 conn(0x7f1bc80f1f50 msgr2=0x7f1bc80fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:00.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.961+0000 7f1bce3b3700 1 -- 172.21.15.78:0/596189319 shutdown_connections 2023-12-21T03:53:00.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:00.961+0000 7f1bce3b3700 1 -- 172.21.15.78:0/596189319 wait complete. 2023-12-21T03:53:01.334 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:01.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:52:59.648460+0000 mgr.smithi078.zidbgq (mgr.14162) 453 : cluster [DBG] pgmap v430: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:01.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:00.960866+0000 mon.smithi078 (mon.0) 504 : audit [DBG] from='client.? 172.21.15.78:0/596189319' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:52:59.648460+0000 mgr.smithi078.zidbgq (mgr.14162) 453 : cluster [DBG] pgmap v430: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:00.960866+0000 mon.smithi078 (mon.0) 504 : audit [DBG] from='client.? 172.21.15.78:0/596189319' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:03.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:01.648752+0000 mgr.smithi078.zidbgq (mgr.14162) 454 : cluster [DBG] pgmap v431: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:01.648752+0000 mgr.smithi078.zidbgq (mgr.14162) 454 : cluster [DBG] pgmap v431: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:04.335 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:04.573 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:05.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:03.649260+0000 mgr.smithi078.zidbgq (mgr.14162) 455 : cluster [DBG] pgmap v432: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:05.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/1103496782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec00f9930 msgr2=0x7f1ec00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:05.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 --2- 172.21.15.78:0/1103496782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec00f9930 0x7f1ec00fbd80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f1ea8004660 tx=0x7f1ea8009b30).stop 2023-12-21T03:53:05.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/1103496782 shutdown_connections 2023-12-21T03:53:05.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 --2- 172.21.15.78:0/1103496782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec00f9930 0x7f1ec00fbd80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:05.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 --2- 172.21.15.78:0/1103496782 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ec0060870 0x7f1ec00f93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:53:05.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/1103496782 >> 172.21.15.78:0/1103496782 conn(0x7f1ec00f0e10 msgr2=0x7f1ec00f3250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:05.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/1103496782 shutdown_connections 2023-12-21T03:53:05.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/1103496782 wait complete. 2023-12-21T03:53:05.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 Processor -- start 2023-12-21T03:53:05.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 -- start start 2023-12-21T03:53:05.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ec0060870 0x7f1ec00f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:05.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec00f9930 0x7f1ec00f52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:05.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00f58a0 con 0x7f1ec00f9930 2023-12-21T03:53:05.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1ec5dd8700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00f59e0 con 0x7f1ec0060870 2023-12-21T03:53:05.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1eb7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec00f9930 0x7f1ec00f52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:05.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1eb7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec00f9930 0x7f1ec00f52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57406/0 (socket says 172.21.15.78:57406) 2023-12-21T03:53:05.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.649+0000 7f1eb7fff700 1 -- 172.21.15.78:0/509976649 learned_addr learned my addr 172.21.15.78:0/509976649 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:05.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.650+0000 7f1ebf7fe700 1 --2- 172.21.15.78:0/509976649 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ec0060870 0x7f1ec00f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:05.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.650+0000 7f1eb7fff700 1 -- 172.21.15.78:0/509976649 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ec0060870 msgr2=0x7f1ec00f6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:05.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.650+0000 7f1eb7fff700 1 --2- 172.21.15.78:0/509976649 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ec0060870 0x7f1ec00f6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:05.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.650+0000 7f1eb7fff700 1 -- 172.21.15.78:0/509976649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8005040 con 0x7f1ec00f9930 2023-12-21T03:53:05.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.650+0000 7f1eb7fff700 1 --2- 172.21.15.78:0/509976649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec00f9930 0x7f1ec00f52b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f1ea8009f50 tx=0x7f1ea8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:05.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.650+0000 7f1ebcff9700 1 -- 172.21.15.78:0/509976649 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ea8018070 con 0x7f1ec00f9930 2023-12-21T03:53:05.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.650+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/509976649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec00f5c60 con 0x7f1ec00f9930 2023-12-21T03:53:05.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.650+0000 7f1ebcff9700 1 -- 172.21.15.78:0/509976649 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ea8013410 con 0x7f1ec00f9930 2023-12-21T03:53:05.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.650+0000 7f1ebcff9700 1 -- 172.21.15.78:0/509976649 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ea801c620 con 0x7f1ec00f9930 2023-12-21T03:53:05.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.650+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/509976649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec010f070 con 0x7f1ec00f9930 2023-12-21T03:53:05.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.651+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/509976649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec0045ad0 con 0x7f1ec00f9930 2023-12-21T03:53:05.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.651+0000 7f1ebcff9700 1 -- 172.21.15.78:0/509976649 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1ea801d420 con 0x7f1ec00f9930 2023-12-21T03:53:05.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.651+0000 7f1ebcff9700 1 --2- 172.21.15.78:0/509976649 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1eac05c2a0 0x7f1eac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:05.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.651+0000 7f1ebcff9700 1 -- 172.21.15.78:0/509976649 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1ea80779f0 con 0x7f1ec00f9930 2023-12-21T03:53:05.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.652+0000 7f1ebf7fe700 1 --2- 172.21.15.78:0/509976649 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1eac05c2a0 0x7f1eac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:05.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.652+0000 7f1ebf7fe700 1 --2- 172.21.15.78:0/509976649 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1eac05c2a0 0x7f1eac05e760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f1ec00f6bc0 tx=0x7f1eb0008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:05.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.655+0000 7f1ebcff9700 1 -- 172.21.15.78:0/509976649 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ea8079e60 con 0x7f1ec00f9930 2023-12-21T03:53:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:03.649260+0000 mgr.smithi078.zidbgq (mgr.14162) 455 : cluster [DBG] pgmap v432: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:05.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.863+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/509976649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1ec0027a00 con 0x7f1ec00f9930 2023-12-21T03:53:05.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.864+0000 7f1ebcff9700 1 -- 172.21.15.78:0/509976649 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f1ea804a9f0 con 0x7f1ec00f9930 2023-12-21T03:53:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:05.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.865+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/509976649 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1eac05c2a0 msgr2=0x7f1eac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:05.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.865+0000 7f1ec5dd8700 1 --2- 172.21.15.78:0/509976649 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1eac05c2a0 0x7f1eac05e760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f1ec00f6bc0 tx=0x7f1eb0008040).stop 2023-12-21T03:53:05.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.865+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/509976649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec00f9930 msgr2=0x7f1ec00f52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:05.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.865+0000 7f1ec5dd8700 1 --2- 172.21.15.78:0/509976649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec00f9930 0x7f1ec00f52b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f1ea8009f50 tx=0x7f1ea8009b30).stop 2023-12-21T03:53:05.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.865+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/509976649 shutdown_connections 2023-12-21T03:53:05.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.865+0000 7f1ec5dd8700 1 --2- 172.21.15.78:0/509976649 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1eac05c2a0 0x7f1eac05e760 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:05.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.865+0000 7f1ec5dd8700 1 --2- 172.21.15.78:0/509976649 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ec0060870 0x7f1ec00f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:05.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.865+0000 7f1ec5dd8700 1 --2- 172.21.15.78:0/509976649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec00f9930 0x7f1ec00f52b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:05.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.865+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/509976649 >> 172.21.15.78:0/509976649 conn(0x7f1ec00f0e10 msgr2=0x7f1ec00f3250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:05.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.866+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/509976649 shutdown_connections 2023-12-21T03:53:05.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:05.866+0000 7f1ec5dd8700 1 -- 172.21.15.78:0/509976649 wait complete. 2023-12-21T03:53:06.234 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:06.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:06 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:05.865021+0000 mon.smithi078 (mon.0) 505 : audit [DBG] from='client.? 172.21.15.78:0/509976649' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:06.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:06 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:05.865021+0000 mon.smithi078 (mon.0) 505 : audit [DBG] from='client.? 172.21.15.78:0/509976649' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:07.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:05.649607+0000 mgr.smithi078.zidbgq (mgr.14162) 456 : cluster [DBG] pgmap v433: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:05.649607+0000 mgr.smithi078.zidbgq (mgr.14162) 456 : cluster [DBG] pgmap v433: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:09.235 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:09.473 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:09.567 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:07.650060+0000 mgr.smithi078.zidbgq (mgr.14162) 457 : cluster [DBG] pgmap v434: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:09.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:07.650060+0000 mgr.smithi078.zidbgq (mgr.14162) 457 : cluster [DBG] pgmap v434: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:10.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.451+0000 7ff7cd878700 1 -- 172.21.15.78:0/931171763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c80f6960 msgr2=0x7ff7c80f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:10.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.451+0000 7ff7cd878700 1 --2- 172.21.15.78:0/931171763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c80f6960 0x7ff7c80f6dc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7ff7b8004660 tx=0x7ff7b8009b30).stop 2023-12-21T03:53:10.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.451+0000 7ff7cd878700 1 -- 172.21.15.78:0/931171763 shutdown_connections 2023-12-21T03:53:10.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.451+0000 7ff7cd878700 1 --2- 172.21.15.78:0/931171763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c80f6960 0x7ff7c80f6dc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.451+0000 7ff7cd878700 1 --2- 172.21.15.78:0/931171763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c80f56c0 0x7ff7c80f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.451+0000 7ff7cd878700 1 -- 172.21.15.78:0/931171763 >> 172.21.15.78:0/931171763 conn(0x7ff7c80f0e20 msgr2=0x7ff7c80f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.451+0000 7ff7cd878700 1 -- 172.21.15.78:0/931171763 shutdown_connections 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.451+0000 7ff7cd878700 1 -- 172.21.15.78:0/931171763 wait complete. 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7cd878700 1 Processor -- start 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7cd878700 1 -- start start 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7cd878700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c80f56c0 0x7ff7c81070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7cd878700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c80f6960 0x7ff7c8107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7cd878700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8107b40 con 0x7ff7c80f56c0 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7cd878700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8107c80 con 0x7ff7c80f6960 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7c6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c80f56c0 0x7ff7c81070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7c6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c80f56c0 0x7ff7c81070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38918/0 (socket says 172.21.15.78:38918) 2023-12-21T03:53:10.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7c6ffd700 1 -- 172.21.15.78:0/1002660518 learned_addr learned my addr 172.21.15.78:0/1002660518 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:10.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7c67fc700 1 --2- 172.21.15.78:0/1002660518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c80f6960 0x7ff7c8107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:10.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7c6ffd700 1 -- 172.21.15.78:0/1002660518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c80f6960 msgr2=0x7ff7c8107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:10.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7c6ffd700 1 --2- 172.21.15.78:0/1002660518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c80f6960 0x7ff7c8107590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:10.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7c6ffd700 1 -- 172.21.15.78:0/1002660518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b8005040 con 0x7ff7c80f56c0 2023-12-21T03:53:10.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7c6ffd700 1 --2- 172.21.15.78:0/1002660518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c80f56c0 0x7ff7c81070a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7ff7b0002700 tx=0x7ff7b0009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:10.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.452+0000 7ff7bf7fe700 1 -- 172.21.15.78:0/1002660518 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7b00079c0 con 0x7ff7c80f56c0 2023-12-21T03:53:10.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.453+0000 7ff7cd878700 1 -- 172.21.15.78:0/1002660518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c8108620 con 0x7ff7c80f56c0 2023-12-21T03:53:10.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.453+0000 7ff7bf7fe700 1 -- 172.21.15.78:0/1002660518 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7b000f040 con 0x7ff7c80f56c0 2023-12-21T03:53:10.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.453+0000 7ff7bf7fe700 1 -- 172.21.15.78:0/1002660518 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7b0014860 con 0x7ff7c80f56c0 2023-12-21T03:53:10.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.453+0000 7ff7cd878700 1 -- 172.21.15.78:0/1002660518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c8108be0 con 0x7ff7c80f56c0 2023-12-21T03:53:10.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.453+0000 7ff7cd878700 1 -- 172.21.15.78:0/1002660518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c8045ad0 con 0x7ff7c80f56c0 2023-12-21T03:53:10.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.454+0000 7ff7bf7fe700 1 -- 172.21.15.78:0/1002660518 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff7b001b060 con 0x7ff7c80f56c0 2023-12-21T03:53:10.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.454+0000 7ff7bf7fe700 1 --2- 172.21.15.78:0/1002660518 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff7b405c2a0 0x7ff7b405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:10.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.454+0000 7ff7bf7fe700 1 -- 172.21.15.78:0/1002660518 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff7b0075a70 con 0x7ff7c80f56c0 2023-12-21T03:53:10.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.454+0000 7ff7c67fc700 1 --2- 172.21.15.78:0/1002660518 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff7b405c2a0 0x7ff7b405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:10.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.455+0000 7ff7c67fc700 1 --2- 172.21.15.78:0/1002660518 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff7b405c2a0 0x7ff7b405e760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff7b8009f50 tx=0x7ff7b8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:10.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.458+0000 7ff7bf7fe700 1 -- 172.21.15.78:0/1002660518 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7b0048a70 con 0x7ff7c80f56c0 2023-12-21T03:53:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.666+0000 7ff7cd878700 1 -- 172.21.15.78:0/1002660518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff7c81092e0 con 0x7ff7c80f56c0 2023-12-21T03:53:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.666+0000 7ff7bf7fe700 1 -- 172.21.15.78:0/1002660518 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7ff7b0075e60 con 0x7ff7c80f56c0 2023-12-21T03:53:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:10.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.668+0000 7ff7cd878700 1 -- 172.21.15.78:0/1002660518 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff7b405c2a0 msgr2=0x7ff7b405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:10.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.668+0000 7ff7cd878700 1 --2- 172.21.15.78:0/1002660518 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff7b405c2a0 0x7ff7b405e760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff7b8009f50 tx=0x7ff7b8009b30).stop 2023-12-21T03:53:10.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.668+0000 7ff7cd878700 1 -- 172.21.15.78:0/1002660518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c80f56c0 msgr2=0x7ff7c81070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:10.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.668+0000 7ff7cd878700 1 --2- 172.21.15.78:0/1002660518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c80f56c0 0x7ff7c81070a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7ff7b0002700 tx=0x7ff7b0009d80).stop 2023-12-21T03:53:10.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.668+0000 7ff7cd878700 1 -- 172.21.15.78:0/1002660518 shutdown_connections 2023-12-21T03:53:10.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.668+0000 7ff7cd878700 1 --2- 172.21.15.78:0/1002660518 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff7b405c2a0 0x7ff7b405e760 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:10.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.668+0000 7ff7cd878700 1 --2- 172.21.15.78:0/1002660518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c80f56c0 0x7ff7c81070a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:10.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.668+0000 7ff7cd878700 1 --2- 172.21.15.78:0/1002660518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c80f6960 0x7ff7c8107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:10.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.668+0000 7ff7cd878700 1 -- 172.21.15.78:0/1002660518 >> 172.21.15.78:0/1002660518 conn(0x7ff7c80f0e20 msgr2=0x7ff7c80603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:10.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.668+0000 7ff7cd878700 1 -- 172.21.15.78:0/1002660518 shutdown_connections 2023-12-21T03:53:10.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:10.668+0000 7ff7cd878700 1 -- 172.21.15.78:0/1002660518 wait complete. 2023-12-21T03:53:11.042 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:09.650405+0000 mgr.smithi078.zidbgq (mgr.14162) 458 : cluster [DBG] pgmap v435: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:10.667370+0000 mon.smithi078 (mon.0) 506 : audit [DBG] from='client.? 172.21.15.78:0/1002660518' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:09.650405+0000 mgr.smithi078.zidbgq (mgr.14162) 458 : cluster [DBG] pgmap v435: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:10.667370+0000 mon.smithi078 (mon.0) 506 : audit [DBG] from='client.? 172.21.15.78:0/1002660518' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:13.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:11.650807+0000 mgr.smithi078.zidbgq (mgr.14162) 459 : cluster [DBG] pgmap v436: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:11.650807+0000 mgr.smithi078.zidbgq (mgr.14162) 459 : cluster [DBG] pgmap v436: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:14.044 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:14.289 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:15.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.215+0000 7f0340281700 1 -- 172.21.15.78:0/1460062360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380f67d0 msgr2=0x7f03380f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:15.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.215+0000 7f0340281700 1 --2- 172.21.15.78:0/1460062360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380f67d0 0x7f03380f6bf0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f0328004660 tx=0x7f0328009b30).stop 2023-12-21T03:53:15.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.215+0000 7f0340281700 1 -- 172.21.15.78:0/1460062360 shutdown_connections 2023-12-21T03:53:15.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.215+0000 7f0340281700 1 --2- 172.21.15.78:0/1460062360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03380f7a70 0x7f03380f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.215+0000 7f0340281700 1 --2- 172.21.15.78:0/1460062360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380f67d0 0x7f03380f6bf0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.215+0000 7f0340281700 1 -- 172.21.15.78:0/1460062360 >> 172.21.15.78:0/1460062360 conn(0x7f03380f1f50 msgr2=0x7f03380f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.215+0000 7f0340281700 1 -- 172.21.15.78:0/1460062360 shutdown_connections 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.215+0000 7f0340281700 1 -- 172.21.15.78:0/1460062360 wait complete. 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f0340281700 1 Processor -- start 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f0340281700 1 -- start start 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f0340281700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380f67d0 0x7f03381070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f0340281700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03380f7a70 0x7f03381075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f0340281700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0338107c10 con 0x7f03380f67d0 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f0340281700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03381084c0 con 0x7f03380f7a70 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f033e01d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380f67d0 0x7f03381070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f033e01d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380f67d0 0x7f03381070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38928/0 (socket says 172.21.15.78:38928) 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f033e01d700 1 -- 172.21.15.78:0/3047297908 learned_addr learned my addr 172.21.15.78:0/3047297908 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:15.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f033e01d700 1 -- 172.21.15.78:0/3047297908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03380f7a70 msgr2=0x7f03381075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:15.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f033e01d700 1 --2- 172.21.15.78:0/3047297908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03380f7a70 0x7f03381075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:53:15.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f033e01d700 1 -- 172.21.15.78:0/3047297908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328005040 con 0x7f03380f67d0 2023-12-21T03:53:15.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f033e01d700 1 --2- 172.21.15.78:0/3047297908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380f67d0 0x7f03381070e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f0328009f50 tx=0x7f0328009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:15.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.216+0000 7f032effd700 1 -- 172.21.15.78:0/3047297908 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0328018070 con 0x7f03380f67d0 2023-12-21T03:53:15.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.217+0000 7f032effd700 1 -- 172.21.15.78:0/3047297908 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0328013410 con 0x7f03380f67d0 2023-12-21T03:53:15.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.217+0000 7f0340281700 1 -- 172.21.15.78:0/3047297908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0338108740 con 0x7f03380f67d0 2023-12-21T03:53:15.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.217+0000 7f032effd700 1 -- 172.21.15.78:0/3047297908 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f032801c620 con 0x7f03380f67d0 2023-12-21T03:53:15.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.217+0000 7f0340281700 1 -- 172.21.15.78:0/3047297908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0338108ca0 con 0x7f03380f67d0 2023-12-21T03:53:15.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.217+0000 7f0340281700 1 -- 172.21.15.78:0/3047297908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0338059100 con 0x7f03380f67d0 2023-12-21T03:53:15.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.218+0000 7f032effd700 1 -- 172.21.15.78:0/3047297908 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f032801d420 con 0x7f03380f67d0 2023-12-21T03:53:15.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.218+0000 7f032effd700 1 --2- 172.21.15.78:0/3047297908 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f032405c2a0 0x7f032405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:15.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.218+0000 7f032effd700 1 -- 172.21.15.78:0/3047297908 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0328077a40 con 0x7f03380f67d0 2023-12-21T03:53:15.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.218+0000 7f033d81c700 1 --2- 172.21.15.78:0/3047297908 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f032405c2a0 0x7f032405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:15.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.218+0000 7f033d81c700 1 --2- 172.21.15.78:0/3047297908 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f032405c2a0 0x7f032405e760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f0334009610 tx=0x7f0334006ab0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:15.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.222+0000 7f032effd700 1 -- 172.21.15.78:0/3047297908 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f032804aa40 con 0x7f03380f67d0 2023-12-21T03:53:15.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.429+0000 7f0340281700 1 -- 172.21.15.78:0/3047297908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f03380ff170 con 0x7f03380f67d0 2023-12-21T03:53:15.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.429+0000 7f032effd700 1 -- 172.21.15.78:0/3047297908 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f03380ff170 con 0x7f03380f67d0 2023-12-21T03:53:15.432 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:15.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.431+0000 7f0340281700 1 -- 172.21.15.78:0/3047297908 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f032405c2a0 msgr2=0x7f032405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:15.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.431+0000 7f0340281700 1 --2- 172.21.15.78:0/3047297908 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f032405c2a0 0x7f032405e760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f0334009610 tx=0x7f0334006ab0).stop 2023-12-21T03:53:15.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.431+0000 7f0340281700 1 -- 172.21.15.78:0/3047297908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380f67d0 msgr2=0x7f03381070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:15.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.431+0000 7f0340281700 1 --2- 172.21.15.78:0/3047297908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380f67d0 0x7f03381070e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f0328009f50 tx=0x7f0328009b30).stop 2023-12-21T03:53:15.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.431+0000 7f0340281700 1 -- 172.21.15.78:0/3047297908 shutdown_connections 2023-12-21T03:53:15.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.431+0000 7f0340281700 1 --2- 172.21.15.78:0/3047297908 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f032405c2a0 0x7f032405e760 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:15.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.431+0000 7f0340281700 1 --2- 172.21.15.78:0/3047297908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380f67d0 0x7f03381070e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:15.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.431+0000 7f0340281700 1 --2- 172.21.15.78:0/3047297908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03380f7a70 0x7f03381075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:53:15.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.431+0000 7f0340281700 1 -- 172.21.15.78:0/3047297908 >> 172.21.15.78:0/3047297908 conn(0x7f03380f1f50 msgr2=0x7f03380fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:15.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.431+0000 7f0340281700 1 -- 172.21.15.78:0/3047297908 shutdown_connections 2023-12-21T03:53:15.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:15.431+0000 7f0340281700 1 -- 172.21.15.78:0/3047297908 wait complete. 2023-12-21T03:53:15.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:13.651188+0000 mgr.smithi078.zidbgq (mgr.14162) 460 : cluster [DBG] pgmap v437: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:15.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:15.430719+0000 mon.smithi078 (mon.0) 507 : audit [DBG] from='client.? 172.21.15.78:0/3047297908' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:13.651188+0000 mgr.smithi078.zidbgq (mgr.14162) 460 : cluster [DBG] pgmap v437: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:15.430719+0000 mon.smithi078 (mon.0) 507 : audit [DBG] from='client.? 172.21.15.78:0/3047297908' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:15.833 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:17.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:15.651514+0000 mgr.smithi078.zidbgq (mgr.14162) 461 : cluster [DBG] pgmap v438: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:17.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:15.651514+0000 mgr.smithi078.zidbgq (mgr.14162) 461 : cluster [DBG] pgmap v438: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:18.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:19.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:17.651886+0000 mgr.smithi078.zidbgq (mgr.14162) 462 : cluster [DBG] pgmap v439: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:19.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:17.651886+0000 mgr.smithi078.zidbgq (mgr.14162) 462 : cluster [DBG] pgmap v439: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.013+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/3136509500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c0f67c0 msgr2=0x7fa89c0f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.013+0000 7fa8a3a4c700 1 --2- 172.21.15.78:0/3136509500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c0f67c0 0x7fa89c0f6be0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fa898004660 tx=0x7fa898009b30).stop 2023-12-21T03:53:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.013+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/3136509500 shutdown_connections 2023-12-21T03:53:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.013+0000 7fa8a3a4c700 1 --2- 172.21.15.78:0/3136509500 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89c0f7a60 0x7fa89c0f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.013+0000 7fa8a3a4c700 1 --2- 172.21.15.78:0/3136509500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c0f67c0 0x7fa89c0f6be0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.013+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/3136509500 >> 172.21.15.78:0/3136509500 conn(0x7fa89c0f1f20 msgr2=0x7fa89c0f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.013+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/3136509500 shutdown_connections 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.013+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/3136509500 wait complete. 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a3a4c700 1 Processor -- start 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a3a4c700 1 -- start start 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a3a4c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c0f67c0 0x7fa89c107030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a3a4c700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89c0f7a60 0x7fa89c107520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a3a4c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c107ad0 con 0x7fa89c0f67c0 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a3a4c700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c107c10 con 0x7fa89c0f7a60 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a17e8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c0f67c0 0x7fa89c107030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a17e8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c0f67c0 0x7fa89c107030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39954/0 (socket says 172.21.15.78:39954) 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a17e8700 1 -- 172.21.15.78:0/104081243 learned_addr learned my addr 172.21.15.78:0/104081243 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a0fe7700 1 --2- 172.21.15.78:0/104081243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89c0f7a60 0x7fa89c107520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a17e8700 1 -- 172.21.15.78:0/104081243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89c0f7a60 msgr2=0x7fa89c107520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a17e8700 1 --2- 172.21.15.78:0/104081243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89c0f7a60 0x7fa89c107520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a17e8700 1 -- 172.21.15.78:0/104081243 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa898005040 con 0x7fa89c0f67c0 2023-12-21T03:53:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa8a17e8700 1 --2- 172.21.15.78:0/104081243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c0f67c0 0x7fa89c107030 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fa898000c00 tx=0x7fa89800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.014+0000 7fa891ffb700 1 -- 172.21.15.78:0/104081243 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa898019070 con 0x7fa89c0f67c0 2023-12-21T03:53:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.015+0000 7fa891ffb700 1 -- 172.21.15.78:0/104081243 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa898007e70 con 0x7fa89c0f67c0 2023-12-21T03:53:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.015+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/104081243 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89c108550 con 0x7fa89c0f67c0 2023-12-21T03:53:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.015+0000 7fa891ffb700 1 -- 172.21.15.78:0/104081243 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa898014db0 con 0x7fa89c0f67c0 2023-12-21T03:53:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.015+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/104081243 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89c108ab0 con 0x7fa89c0f67c0 2023-12-21T03:53:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.016+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/104081243 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89c045ad0 con 0x7fa89c0f67c0 2023-12-21T03:53:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.016+0000 7fa891ffb700 1 -- 172.21.15.78:0/104081243 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa898025070 con 0x7fa89c0f67c0 2023-12-21T03:53:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.016+0000 7fa891ffb700 1 --2- 172.21.15.78:0/104081243 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa88805c2a0 0x7fa88805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.017+0000 7fa891ffb700 1 -- 172.21.15.78:0/104081243 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa898076a70 con 0x7fa89c0f67c0 2023-12-21T03:53:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.017+0000 7fa8a0fe7700 1 --2- 172.21.15.78:0/104081243 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa88805c2a0 0x7fa88805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.017+0000 7fa8a0fe7700 1 --2- 172.21.15.78:0/104081243 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa88805c2a0 0x7fa88805e760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fa88c007f80 tx=0x7fa88c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:20.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.020+0000 7fa891ffb700 1 -- 172.21.15.78:0/104081243 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa898049af0 con 0x7fa89c0f67c0 2023-12-21T03:53:20.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.229+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/104081243 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa89c027a00 con 0x7fa89c0f67c0 2023-12-21T03:53:20.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.229+0000 7fa891ffb700 1 -- 172.21.15.78:0/104081243 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fa898076e60 con 0x7fa89c0f67c0 2023-12-21T03:53:20.231 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:20.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.231+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/104081243 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa88805c2a0 msgr2=0x7fa88805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:20.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.231+0000 7fa8a3a4c700 1 --2- 172.21.15.78:0/104081243 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa88805c2a0 0x7fa88805e760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fa88c007f80 tx=0x7fa88c00b040).stop 2023-12-21T03:53:20.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.231+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/104081243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c0f67c0 msgr2=0x7fa89c107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:20.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.231+0000 7fa8a3a4c700 1 --2- 172.21.15.78:0/104081243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c0f67c0 0x7fa89c107030 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fa898000c00 tx=0x7fa89800b040).stop 2023-12-21T03:53:20.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.231+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/104081243 shutdown_connections 2023-12-21T03:53:20.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.231+0000 7fa8a3a4c700 1 --2- 172.21.15.78:0/104081243 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa88805c2a0 0x7fa88805e760 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:20.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.231+0000 7fa8a3a4c700 1 --2- 172.21.15.78:0/104081243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c0f67c0 0x7fa89c107030 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:20.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.231+0000 7fa8a3a4c700 1 --2- 172.21.15.78:0/104081243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89c0f7a60 0x7fa89c107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:20.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.231+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/104081243 >> 172.21.15.78:0/104081243 conn(0x7fa89c0f1f20 msgr2=0x7fa89c0fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:20.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.231+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/104081243 shutdown_connections 2023-12-21T03:53:20.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:20.231+0000 7fa8a3a4c700 1 -- 172.21.15.78:0/104081243 wait complete. 2023-12-21T03:53:20.539 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:20 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:20.230753+0000 mon.smithi078 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.78:0/104081243' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:20.598 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:20.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:20 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:20.230753+0000 mon.smithi078 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.78:0/104081243' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:21.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:19.652493+0000 mgr.smithi078.zidbgq (mgr.14162) 463 : cluster [DBG] pgmap v440: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:19.652493+0000 mgr.smithi078.zidbgq (mgr.14162) 463 : cluster [DBG] pgmap v440: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:23.599 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:23.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:21.652936+0000 mgr.smithi078.zidbgq (mgr.14162) 464 : cluster [DBG] pgmap v441: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:21.652936+0000 mgr.smithi078.zidbgq (mgr.14162) 464 : cluster [DBG] pgmap v441: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:23.838 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 -- 172.21.15.78:0/274495091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83380f56c0 msgr2=0x7f83380f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 --2- 172.21.15.78:0/274495091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83380f56c0 0x7f83380f5ae0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f8328004660 tx=0x7f8328009b30).stop 2023-12-21T03:53:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 -- 172.21.15.78:0/274495091 shutdown_connections 2023-12-21T03:53:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 --2- 172.21.15.78:0/274495091 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83380f6960 0x7f83380f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 --2- 172.21.15.78:0/274495091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83380f56c0 0x7f83380f5ae0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 -- 172.21.15.78:0/274495091 >> 172.21.15.78:0/274495091 conn(0x7f83380f0e20 msgr2=0x7f83380f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 -- 172.21.15.78:0/274495091 shutdown_connections 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 -- 172.21.15.78:0/274495091 wait complete. 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 Processor -- start 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 -- start start 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83380f56c0 0x7f83380f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83380f6960 0x7f83380f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380f8200 con 0x7f83380f56c0 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833f4bc700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380f8340 con 0x7f83380f6960 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833d258700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83380f56c0 0x7f83380f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833d258700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83380f56c0 0x7f83380f9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39966/0 (socket says 172.21.15.78:39966) 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.788+0000 7f833d258700 1 -- 172.21.15.78:0/1398947754 learned_addr learned my addr 172.21.15.78:0/1398947754 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.789+0000 7f833ca57700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83380f6960 0x7f83380f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.789+0000 7f833d258700 1 -- 172.21.15.78:0/1398947754 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83380f6960 msgr2=0x7f83380f7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.789+0000 7f833d258700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83380f6960 0x7f83380f7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.789+0000 7f833d258700 1 -- 172.21.15.78:0/1398947754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8328005040 con 0x7f83380f56c0 2023-12-21T03:53:24.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.789+0000 7f833ca57700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83380f6960 0x7f83380f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:53:24.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.789+0000 7f833d258700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83380f56c0 0x7f83380f9820 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f8328009f50 tx=0x7f8328009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:24.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.789+0000 7f832dffb700 1 -- 172.21.15.78:0/1398947754 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8328018070 con 0x7f83380f56c0 2023-12-21T03:53:24.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.789+0000 7f832dffb700 1 -- 172.21.15.78:0/1398947754 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8328013410 con 0x7f83380f56c0 2023-12-21T03:53:24.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.789+0000 7f833f4bc700 1 -- 172.21.15.78:0/1398947754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83380f85c0 con 0x7f83380f56c0 2023-12-21T03:53:24.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.789+0000 7f832dffb700 1 -- 172.21.15.78:0/1398947754 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f832801c620 con 0x7f83380f56c0 2023-12-21T03:53:24.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.789+0000 7f833f4bc700 1 -- 172.21.15.78:0/1398947754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83380f8b20 con 0x7f83380f56c0 2023-12-21T03:53:24.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.790+0000 7f833f4bc700 1 -- 172.21.15.78:0/1398947754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8338045ad0 con 0x7f83380f56c0 2023-12-21T03:53:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.790+0000 7f832dffb700 1 -- 172.21.15.78:0/1398947754 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f832801d420 con 0x7f83380f56c0 2023-12-21T03:53:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.791+0000 7f832dffb700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f832405c2a0 0x7f832405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.791+0000 7f832dffb700 1 -- 172.21.15.78:0/1398947754 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f83280779f0 con 0x7f83380f56c0 2023-12-21T03:53:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.791+0000 7f833ca57700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f832405c2a0 0x7f832405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.791+0000 7f833ca57700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f832405c2a0 0x7f832405e760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f8334004ac0 tx=0x7f8334009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:24.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:24.795+0000 7f832dffb700 1 -- 172.21.15.78:0/1398947754 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8328079e60 con 0x7f83380f56c0 2023-12-21T03:53:25.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.003+0000 7f833f4bc700 1 -- 172.21.15.78:0/1398947754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f83380fde50 con 0x7f83380f56c0 2023-12-21T03:53:25.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.003+0000 7f832dffb700 1 -- 172.21.15.78:0/1398947754 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f832804a9f0 con 0x7f83380f56c0 2023-12-21T03:53:25.005 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:25.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.005+0000 7f833f4bc700 1 -- 172.21.15.78:0/1398947754 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f832405c2a0 msgr2=0x7f832405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:25.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.005+0000 7f833f4bc700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f832405c2a0 0x7f832405e760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f8334004ac0 tx=0x7f8334009040).stop 2023-12-21T03:53:25.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.005+0000 7f833f4bc700 1 -- 172.21.15.78:0/1398947754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83380f56c0 msgr2=0x7f83380f9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:25.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.005+0000 7f833f4bc700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83380f56c0 0x7f83380f9820 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f8328009f50 tx=0x7f8328009b30).stop 2023-12-21T03:53:25.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.005+0000 7f833f4bc700 1 -- 172.21.15.78:0/1398947754 shutdown_connections 2023-12-21T03:53:25.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.005+0000 7f833f4bc700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f832405c2a0 0x7f832405e760 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:25.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.005+0000 7f833f4bc700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83380f56c0 0x7f83380f9820 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:25.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.005+0000 7f833f4bc700 1 --2- 172.21.15.78:0/1398947754 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83380f6960 0x7f83380f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:25.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.005+0000 7f833f4bc700 1 -- 172.21.15.78:0/1398947754 >> 172.21.15.78:0/1398947754 conn(0x7f83380f0e20 msgr2=0x7f83380603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:25.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.005+0000 7f833f4bc700 1 -- 172.21.15.78:0/1398947754 shutdown_connections 2023-12-21T03:53:25.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:25.005+0000 7f833f4bc700 1 -- 172.21.15.78:0/1398947754 wait complete. 2023-12-21T03:53:25.405 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:23.653292+0000 mgr.smithi078.zidbgq (mgr.14162) 465 : cluster [DBG] pgmap v442: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:25.004908+0000 mon.smithi078 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.78:0/1398947754' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:23.653292+0000 mgr.smithi078.zidbgq (mgr.14162) 465 : cluster [DBG] pgmap v442: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:25.004908+0000 mon.smithi078 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.78:0/1398947754' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:25.653611+0000 mgr.smithi078.zidbgq (mgr.14162) 466 : cluster [DBG] pgmap v443: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:25.653611+0000 mgr.smithi078.zidbgq (mgr.14162) 466 : cluster [DBG] pgmap v443: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:28.407 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:28.638 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:29.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.582+0000 7f734ce26700 1 -- 172.21.15.78:0/4039066108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73480f56c0 msgr2=0x7f73480f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:29.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.582+0000 7f734ce26700 1 --2- 172.21.15.78:0/4039066108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73480f56c0 0x7f73480f5ae0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f7330004660 tx=0x7f7330009b30).stop 2023-12-21T03:53:29.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.582+0000 7f734ce26700 1 -- 172.21.15.78:0/4039066108 shutdown_connections 2023-12-21T03:53:29.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.582+0000 7f734ce26700 1 --2- 172.21.15.78:0/4039066108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73480f6960 0x7f73480f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.582+0000 7f734ce26700 1 --2- 172.21.15.78:0/4039066108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73480f56c0 0x7f73480f5ae0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.582+0000 7f734ce26700 1 -- 172.21.15.78:0/4039066108 >> 172.21.15.78:0/4039066108 conn(0x7f73480f0e20 msgr2=0x7f73480f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.582+0000 7f734ce26700 1 -- 172.21.15.78:0/4039066108 shutdown_connections 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.582+0000 7f734ce26700 1 -- 172.21.15.78:0/4039066108 wait complete. 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.582+0000 7f734ce26700 1 Processor -- start 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.582+0000 7f734ce26700 1 -- start start 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f734ce26700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73480f56c0 0x7f73481070f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f734ce26700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73480f6960 0x7f73481075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f734ce26700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7348107b90 con 0x7f73480f6960 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f734ce26700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7348107cd0 con 0x7f73480f56c0 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f7345d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73480f6960 0x7f73481075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f7345d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73480f6960 0x7f73481075e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42290/0 (socket says 172.21.15.78:42290) 2023-12-21T03:53:29.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f7345d9b700 1 -- 172.21.15.78:0/762167000 learned_addr learned my addr 172.21.15.78:0/762167000 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f734659c700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73480f56c0 0x7f73481070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f7345d9b700 1 -- 172.21.15.78:0/762167000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73480f56c0 msgr2=0x7f73481070f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f7345d9b700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73480f56c0 0x7f73481070f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f7345d9b700 1 -- 172.21.15.78:0/762167000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7330005040 con 0x7f73480f6960 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f734659c700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73480f56c0 0x7f73481070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f7345d9b700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73480f6960 0x7f73481075e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f73380026d0 tx=0x7f7338009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f733effd700 1 -- 172.21.15.78:0/762167000 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7338009320 con 0x7f73480f6960 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f734ce26700 1 -- 172.21.15.78:0/762167000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7348108670 con 0x7f73480f6960 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f733effd700 1 -- 172.21.15.78:0/762167000 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f733800f040 con 0x7f73480f6960 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f733effd700 1 -- 172.21.15.78:0/762167000 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73380147e0 con 0x7f73480f6960 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.583+0000 7f734ce26700 1 -- 172.21.15.78:0/762167000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7348108c30 con 0x7f73480f6960 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.584+0000 7f734ce26700 1 -- 172.21.15.78:0/762167000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7348045ad0 con 0x7f73480f6960 2023-12-21T03:53:29.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.584+0000 7f733effd700 1 -- 172.21.15.78:0/762167000 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7338009480 con 0x7f73480f6960 2023-12-21T03:53:29.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.585+0000 7f733effd700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f733405c2a0 0x7f733405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:29.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.585+0000 7f733effd700 1 -- 172.21.15.78:0/762167000 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7338076270 con 0x7f73480f6960 2023-12-21T03:53:29.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.585+0000 7f734659c700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f733405c2a0 0x7f733405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:29.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.585+0000 7f734659c700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f733405c2a0 0x7f733405e760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f7330004e30 tx=0x7f7330009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:29.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.589+0000 7f733effd700 1 -- 172.21.15.78:0/762167000 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7338049270 con 0x7f73480f6960 2023-12-21T03:53:29.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:27.653967+0000 mgr.smithi078.zidbgq (mgr.14162) 467 : cluster [DBG] pgmap v444: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:29.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.798+0000 7f734ce26700 1 -- 172.21.15.78:0/762167000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f73480fdf00 con 0x7f73480f6960 2023-12-21T03:53:29.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.798+0000 7f733effd700 1 -- 172.21.15.78:0/762167000 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f733804c890 con 0x7f73480f6960 2023-12-21T03:53:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:29.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.799+0000 7f734ce26700 1 -- 172.21.15.78:0/762167000 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f733405c2a0 msgr2=0x7f733405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:29.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.799+0000 7f734ce26700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f733405c2a0 0x7f733405e760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f7330004e30 tx=0x7f7330009b30).stop 2023-12-21T03:53:29.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.800+0000 7f734ce26700 1 -- 172.21.15.78:0/762167000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73480f6960 msgr2=0x7f73481075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:29.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.800+0000 7f734ce26700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73480f6960 0x7f73481075e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f73380026d0 tx=0x7f7338009de0).stop 2023-12-21T03:53:29.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.800+0000 7f734ce26700 1 -- 172.21.15.78:0/762167000 shutdown_connections 2023-12-21T03:53:29.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.800+0000 7f734ce26700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f733405c2a0 0x7f733405e760 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:29.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.800+0000 7f734ce26700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73480f56c0 0x7f73481070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:29.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.800+0000 7f734ce26700 1 --2- 172.21.15.78:0/762167000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73480f6960 0x7f73481075e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:29.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.800+0000 7f734ce26700 1 -- 172.21.15.78:0/762167000 >> 172.21.15.78:0/762167000 conn(0x7f73480f0e20 msgr2=0x7f734805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:29.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.800+0000 7f734ce26700 1 -- 172.21.15.78:0/762167000 shutdown_connections 2023-12-21T03:53:29.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:29.800+0000 7f734ce26700 1 -- 172.21.15.78:0/762167000 wait complete. 2023-12-21T03:53:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:27.653967+0000 mgr.smithi078.zidbgq (mgr.14162) 467 : cluster [DBG] pgmap v444: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:30.173 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:30.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:29.799449+0000 mon.smithi078 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.78:0/762167000' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:30.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:29.799449+0000 mon.smithi078 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.78:0/762167000' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:29.654275+0000 mgr.smithi078.zidbgq (mgr.14162) 468 : cluster [DBG] pgmap v445: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:30.566785+0000 mon.smithi078 (mon.0) 511 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:53:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:30.566951+0000 mon.smithi078 (mon.0) 512 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:53:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:29.654275+0000 mgr.smithi078.zidbgq (mgr.14162) 468 : cluster [DBG] pgmap v445: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:30.566785+0000 mon.smithi078 (mon.0) 511 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:53:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:30.566951+0000 mon.smithi078 (mon.0) 512 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:53:33.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:33.406 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:31.654679+0000 mgr.smithi078.zidbgq (mgr.14162) 469 : cluster [DBG] pgmap v446: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:33.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:31.654679+0000 mgr.smithi078.zidbgq (mgr.14162) 469 : cluster [DBG] pgmap v446: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:34.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.311+0000 7efdb9818700 1 -- 172.21.15.78:0/3031165146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdb4060870 msgr2=0x7efdb40f93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:34.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.311+0000 7efdb9818700 1 --2- 172.21.15.78:0/3031165146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdb4060870 0x7efdb40f93c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7efda4004660 tx=0x7efda4009b30).stop 2023-12-21T03:53:34.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.311+0000 7efdb9818700 1 -- 172.21.15.78:0/3031165146 shutdown_connections 2023-12-21T03:53:34.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.311+0000 7efdb9818700 1 --2- 172.21.15.78:0/3031165146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb40f9940 0x7efdb40fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.311+0000 7efdb9818700 1 --2- 172.21.15.78:0/3031165146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdb4060870 0x7efdb40f93c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.311+0000 7efdb9818700 1 -- 172.21.15.78:0/3031165146 >> 172.21.15.78:0/3031165146 conn(0x7efdb40f0e20 msgr2=0x7efdb40f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.311+0000 7efdb9818700 1 -- 172.21.15.78:0/3031165146 shutdown_connections 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.311+0000 7efdb9818700 1 -- 172.21.15.78:0/3031165146 wait complete. 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.312+0000 7efdb9818700 1 Processor -- start 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.312+0000 7efdb9818700 1 -- start start 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.312+0000 7efdb9818700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdb4060870 0x7efdb41070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.312+0000 7efdb9818700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb40f9940 0x7efdb41075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.312+0000 7efdb9818700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb4107bf0 con 0x7efdb4060870 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.312+0000 7efdb9818700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb41084a0 con 0x7efdb40f9940 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.312+0000 7efdb2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdb4060870 0x7efdb41070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.312+0000 7efdb2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdb4060870 0x7efdb41070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42310/0 (socket says 172.21.15.78:42310) 2023-12-21T03:53:34.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.312+0000 7efdb2ffd700 1 -- 172.21.15.78:0/1079902891 learned_addr learned my addr 172.21.15.78:0/1079902891 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:34.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.312+0000 7efdb27fc700 1 --2- 172.21.15.78:0/1079902891 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb40f9940 0x7efdb41075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:34.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.313+0000 7efdb2ffd700 1 -- 172.21.15.78:0/1079902891 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb40f9940 msgr2=0x7efdb41075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:34.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.313+0000 7efdb2ffd700 1 --2- 172.21.15.78:0/1079902891 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb40f9940 0x7efdb41075b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:34.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.313+0000 7efdb2ffd700 1 -- 172.21.15.78:0/1079902891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda4005040 con 0x7efdb4060870 2023-12-21T03:53:34.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.313+0000 7efdb2ffd700 1 --2- 172.21.15.78:0/1079902891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdb4060870 0x7efdb41070c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7efda4009830 tx=0x7efda4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:34.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.313+0000 7efd9b7fe700 1 -- 172.21.15.78:0/1079902891 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efda4018070 con 0x7efdb4060870 2023-12-21T03:53:34.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.313+0000 7efdb9818700 1 -- 172.21.15.78:0/1079902891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb4108720 con 0x7efdb4060870 2023-12-21T03:53:34.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.313+0000 7efd9b7fe700 1 -- 172.21.15.78:0/1079902891 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efda4013410 con 0x7efdb4060870 2023-12-21T03:53:34.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.313+0000 7efd9b7fe700 1 -- 172.21.15.78:0/1079902891 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efda401c620 con 0x7efdb4060870 2023-12-21T03:53:34.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.313+0000 7efdb9818700 1 -- 172.21.15.78:0/1079902891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb4108c80 con 0x7efdb4060870 2023-12-21T03:53:34.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.314+0000 7efdb9818700 1 -- 172.21.15.78:0/1079902891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb4059100 con 0x7efdb4060870 2023-12-21T03:53:34.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.314+0000 7efd9b7fe700 1 -- 172.21.15.78:0/1079902891 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efda401c8e0 con 0x7efdb4060870 2023-12-21T03:53:34.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.314+0000 7efd9b7fe700 1 --2- 172.21.15.78:0/1079902891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efd9c05c2a0 0x7efd9c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:34.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.315+0000 7efd9b7fe700 1 -- 172.21.15.78:0/1079902891 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7efda4077a60 con 0x7efdb4060870 2023-12-21T03:53:34.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.315+0000 7efdb27fc700 1 --2- 172.21.15.78:0/1079902891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efd9c05c2a0 0x7efd9c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:34.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.315+0000 7efdb27fc700 1 --2- 172.21.15.78:0/1079902891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efd9c05c2a0 0x7efd9c05e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7efdb4060640 tx=0x7efda8008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:34.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.318+0000 7efd9b7fe700 1 -- 172.21.15.78:0/1079902891 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efda404aa00 con 0x7efdb4060870 2023-12-21T03:53:34.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.528+0000 7efdb9818700 1 -- 172.21.15.78:0/1079902891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efdb4027a00 con 0x7efdb4060870 2023-12-21T03:53:34.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.528+0000 7efd9b7fe700 1 -- 172.21.15.78:0/1079902891 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7efda404e020 con 0x7efdb4060870 2023-12-21T03:53:34.530 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:34.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.530+0000 7efdb9818700 1 -- 172.21.15.78:0/1079902891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efd9c05c2a0 msgr2=0x7efd9c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:34.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.530+0000 7efdb9818700 1 --2- 172.21.15.78:0/1079902891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efd9c05c2a0 0x7efd9c05e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7efdb4060640 tx=0x7efda8008040).stop 2023-12-21T03:53:34.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.530+0000 7efdb9818700 1 -- 172.21.15.78:0/1079902891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdb4060870 msgr2=0x7efdb41070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:34.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.530+0000 7efdb9818700 1 --2- 172.21.15.78:0/1079902891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdb4060870 0x7efdb41070c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7efda4009830 tx=0x7efda4009b30).stop 2023-12-21T03:53:34.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.530+0000 7efdb9818700 1 -- 172.21.15.78:0/1079902891 shutdown_connections 2023-12-21T03:53:34.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.530+0000 7efdb9818700 1 --2- 172.21.15.78:0/1079902891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efd9c05c2a0 0x7efd9c05e760 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:34.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.530+0000 7efdb9818700 1 --2- 172.21.15.78:0/1079902891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdb4060870 0x7efdb41070c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.530+0000 7efdb9818700 1 --2- 172.21.15.78:0/1079902891 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb40f9940 0x7efdb41075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.530+0000 7efdb9818700 1 -- 172.21.15.78:0/1079902891 >> 172.21.15.78:0/1079902891 conn(0x7efdb40f0e20 msgr2=0x7efdb40f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.530+0000 7efdb9818700 1 -- 172.21.15.78:0/1079902891 shutdown_connections 2023-12-21T03:53:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:34.530+0000 7efdb9818700 1 -- 172.21.15.78:0/1079902891 wait complete. 2023-12-21T03:53:34.905 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:33.654986+0000 mgr.smithi078.zidbgq (mgr.14162) 470 : cluster [DBG] pgmap v447: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:34.529527+0000 mon.smithi078 (mon.0) 513 : audit [DBG] from='client.? 172.21.15.78:0/1079902891' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:35.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:33.654986+0000 mgr.smithi078.zidbgq (mgr.14162) 470 : cluster [DBG] pgmap v447: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:35.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:34.529527+0000 mon.smithi078 (mon.0) 513 : audit [DBG] from='client.? 172.21.15.78:0/1079902891' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:37.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:35.655313+0000 mgr.smithi078.zidbgq (mgr.14162) 471 : cluster [DBG] pgmap v448: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:35.655313+0000 mgr.smithi078.zidbgq (mgr.14162) 471 : cluster [DBG] pgmap v448: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:37.906 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:38.141 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.106+0000 7f04225fd700 1 -- 172.21.15.78:0/433858726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041c0f6960 msgr2=0x7f041c0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.106+0000 7f04225fd700 1 --2- 172.21.15.78:0/433858726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041c0f6960 0x7f041c0f6dc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f0404004660 tx=0x7f0404009b30).stop 2023-12-21T03:53:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.106+0000 7f04225fd700 1 -- 172.21.15.78:0/433858726 shutdown_connections 2023-12-21T03:53:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.106+0000 7f04225fd700 1 --2- 172.21.15.78:0/433858726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041c0f6960 0x7f041c0f6dc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.106+0000 7f04225fd700 1 --2- 172.21.15.78:0/433858726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f041c0f56c0 0x7f041c0f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.106+0000 7f04225fd700 1 -- 172.21.15.78:0/433858726 >> 172.21.15.78:0/433858726 conn(0x7f041c0f0e20 msgr2=0x7f041c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.106+0000 7f04225fd700 1 -- 172.21.15.78:0/433858726 shutdown_connections 2023-12-21T03:53:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.106+0000 7f04225fd700 1 -- 172.21.15.78:0/433858726 wait complete. 2023-12-21T03:53:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f04225fd700 1 Processor -- start 2023-12-21T03:53:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f04225fd700 1 -- start start 2023-12-21T03:53:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f04225fd700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f041c0f56c0 0x7f041c109de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f04225fd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041c0f6960 0x7f041c10c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f04225fd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c060940 con 0x7f041c0f6960 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f04225fd700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c060a80 con 0x7f041c0f56c0 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f0413fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041c0f6960 0x7f041c10c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f0413fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041c0f6960 0x7f041c10c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35936/0 (socket says 172.21.15.78:35936) 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f0413fff700 1 -- 172.21.15.78:0/1435227810 learned_addr learned my addr 172.21.15.78:0/1435227810 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f041bfff700 1 --2- 172.21.15.78:0/1435227810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f041c0f56c0 0x7f041c109de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f0413fff700 1 -- 172.21.15.78:0/1435227810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f041c0f56c0 msgr2=0x7f041c109de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f0413fff700 1 --2- 172.21.15.78:0/1435227810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f041c0f56c0 0x7f041c109de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f0413fff700 1 -- 172.21.15.78:0/1435227810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0404005040 con 0x7f041c0f6960 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.107+0000 7f0413fff700 1 --2- 172.21.15.78:0/1435227810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041c0f6960 0x7f041c10c2e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f0404009f50 tx=0x7f0404009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.108+0000 7f04197fa700 1 -- 172.21.15.78:0/1435227810 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0404018070 con 0x7f041c0f6960 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.108+0000 7f04197fa700 1 -- 172.21.15.78:0/1435227810 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0404013410 con 0x7f041c0f6960 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.108+0000 7f04225fd700 1 -- 172.21.15.78:0/1435227810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f041c10c830 con 0x7f041c0f6960 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.108+0000 7f04197fa700 1 -- 172.21.15.78:0/1435227810 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f040401c3c0 con 0x7f041c0f6960 2023-12-21T03:53:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.108+0000 7f04225fd700 1 -- 172.21.15.78:0/1435227810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f041c10cd90 con 0x7f041c0f6960 2023-12-21T03:53:39.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.109+0000 7f04225fd700 1 -- 172.21.15.78:0/1435227810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041c045ad0 con 0x7f041c0f6960 2023-12-21T03:53:39.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.109+0000 7f04197fa700 1 -- 172.21.15.78:0/1435227810 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f040401d420 con 0x7f041c0f6960 2023-12-21T03:53:39.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.109+0000 7f04197fa700 1 --2- 172.21.15.78:0/1435227810 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f040805c2a0 0x7f040805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:39.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.109+0000 7f04197fa700 1 -- 172.21.15.78:0/1435227810 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0404077750 con 0x7f041c0f6960 2023-12-21T03:53:39.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.109+0000 7f041bfff700 1 --2- 172.21.15.78:0/1435227810 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f040805c2a0 0x7f040805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:39.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.110+0000 7f041bfff700 1 --2- 172.21.15.78:0/1435227810 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f040805c2a0 0x7f040805e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f041c0605c0 tx=0x7f040c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:39.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.113+0000 7f04197fa700 1 -- 172.21.15.78:0/1435227810 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f040404a7d0 con 0x7f041c0f6960 2023-12-21T03:53:39.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.323+0000 7f04225fd700 1 -- 172.21.15.78:0/1435227810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f041c0fdef0 con 0x7f041c0f6960 2023-12-21T03:53:39.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.323+0000 7f04197fa700 1 -- 172.21.15.78:0/1435227810 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f040404ddf0 con 0x7f041c0f6960 2023-12-21T03:53:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:39.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.325+0000 7f04225fd700 1 -- 172.21.15.78:0/1435227810 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f040805c2a0 msgr2=0x7f040805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:39.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.325+0000 7f04225fd700 1 --2- 172.21.15.78:0/1435227810 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f040805c2a0 0x7f040805e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f041c0605c0 tx=0x7f040c00b040).stop 2023-12-21T03:53:39.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.325+0000 7f04225fd700 1 -- 172.21.15.78:0/1435227810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041c0f6960 msgr2=0x7f041c10c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:39.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.325+0000 7f04225fd700 1 --2- 172.21.15.78:0/1435227810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041c0f6960 0x7f041c10c2e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f0404009f50 tx=0x7f0404009b30).stop 2023-12-21T03:53:39.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.325+0000 7f04225fd700 1 -- 172.21.15.78:0/1435227810 shutdown_connections 2023-12-21T03:53:39.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.325+0000 7f04225fd700 1 --2- 172.21.15.78:0/1435227810 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f040805c2a0 0x7f040805e760 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:39.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.325+0000 7f04225fd700 1 --2- 172.21.15.78:0/1435227810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f041c0f56c0 0x7f041c109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:39.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.325+0000 7f04225fd700 1 --2- 172.21.15.78:0/1435227810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041c0f6960 0x7f041c10c2e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:39.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.325+0000 7f04225fd700 1 -- 172.21.15.78:0/1435227810 >> 172.21.15.78:0/1435227810 conn(0x7f041c0f0e20 msgr2=0x7f041c0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:39.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.325+0000 7f04225fd700 1 -- 172.21.15.78:0/1435227810 shutdown_connections 2023-12-21T03:53:39.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:39.325+0000 7f04225fd700 1 -- 172.21.15.78:0/1435227810 wait complete. 2023-12-21T03:53:39.688 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:37.655757+0000 mgr.smithi078.zidbgq (mgr.14162) 472 : cluster [DBG] pgmap v449: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:39.688 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:39.323819+0000 mon.smithi078 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.78:0/1435227810' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:40.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:37.655757+0000 mgr.smithi078.zidbgq (mgr.14162) 472 : cluster [DBG] pgmap v449: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:40.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:39.323819+0000 mon.smithi078 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.78:0/1435227810' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:39.656158+0000 mgr.smithi078.zidbgq (mgr.14162) 473 : cluster [DBG] pgmap v450: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:42.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:39.656158+0000 mgr.smithi078.zidbgq (mgr.14162) 473 : cluster [DBG] pgmap v450: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:42.743 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:42.981 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:43.800 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:41.656591+0000 mgr.smithi078.zidbgq (mgr.14162) 474 : cluster [DBG] pgmap v451: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:43.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.976+0000 7fcd7ce99700 1 -- 172.21.15.78:0/3134387489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780f67d0 msgr2=0x7fcd780f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:43.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.976+0000 7fcd7ce99700 1 --2- 172.21.15.78:0/3134387489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780f67d0 0x7fcd780f6bf0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fcd68004660 tx=0x7fcd68009b30).stop 2023-12-21T03:53:43.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.976+0000 7fcd7ce99700 1 -- 172.21.15.78:0/3134387489 shutdown_connections 2023-12-21T03:53:43.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.976+0000 7fcd7ce99700 1 --2- 172.21.15.78:0/3134387489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd780f7a70 0x7fcd780f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:43.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.976+0000 7fcd7ce99700 1 --2- 172.21.15.78:0/3134387489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780f67d0 0x7fcd780f6bf0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:43.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.976+0000 7fcd7ce99700 1 -- 172.21.15.78:0/3134387489 >> 172.21.15.78:0/3134387489 conn(0x7fcd780f1f50 msgr2=0x7fcd780f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:43.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd7ce99700 1 -- 172.21.15.78:0/3134387489 shutdown_connections 2023-12-21T03:53:43.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd7ce99700 1 -- 172.21.15.78:0/3134387489 wait complete. 2023-12-21T03:53:43.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd7ce99700 1 Processor -- start 2023-12-21T03:53:43.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd7ce99700 1 -- start start 2023-12-21T03:53:43.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd7ce99700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780f67d0 0x7fcd781070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:43.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd7ce99700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd780f7a70 0x7fcd781075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:43.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd7ce99700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78107b50 con 0x7fcd780f67d0 2023-12-21T03:53:43.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd7ce99700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78107c90 con 0x7fcd780f7a70 2023-12-21T03:53:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd7659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780f67d0 0x7fcd781070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd7659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780f67d0 0x7fcd781070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35958/0 (socket says 172.21.15.78:35958) 2023-12-21T03:53:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd7659c700 1 -- 172.21.15.78:0/473925840 learned_addr learned my addr 172.21.15.78:0/473925840 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.977+0000 7fcd75d9b700 1 --2- 172.21.15.78:0/473925840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd780f7a70 0x7fcd781075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.978+0000 7fcd7659c700 1 -- 172.21.15.78:0/473925840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd780f7a70 msgr2=0x7fcd781075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.978+0000 7fcd7659c700 1 --2- 172.21.15.78:0/473925840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd780f7a70 0x7fcd781075a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.978+0000 7fcd7659c700 1 -- 172.21.15.78:0/473925840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68005040 con 0x7fcd780f67d0 2023-12-21T03:53:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.978+0000 7fcd7659c700 1 --2- 172.21.15.78:0/473925840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780f67d0 0x7fcd781070b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fcd68004e30 tx=0x7fcd68009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.978+0000 7fcd66ffd700 1 -- 172.21.15.78:0/473925840 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd68018070 con 0x7fcd780f67d0 2023-12-21T03:53:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.978+0000 7fcd66ffd700 1 -- 172.21.15.78:0/473925840 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd68013410 con 0x7fcd780f67d0 2023-12-21T03:53:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.978+0000 7fcd7ce99700 1 -- 172.21.15.78:0/473925840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd781085d0 con 0x7fcd780f67d0 2023-12-21T03:53:43.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.978+0000 7fcd66ffd700 1 -- 172.21.15.78:0/473925840 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd6801de20 con 0x7fcd780f67d0 2023-12-21T03:53:43.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.978+0000 7fcd7ce99700 1 -- 172.21.15.78:0/473925840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd78108b30 con 0x7fcd780f67d0 2023-12-21T03:53:43.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.979+0000 7fcd7ce99700 1 -- 172.21.15.78:0/473925840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd78045ad0 con 0x7fcd780f67d0 2023-12-21T03:53:43.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.979+0000 7fcd66ffd700 1 -- 172.21.15.78:0/473925840 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcd6801d420 con 0x7fcd780f67d0 2023-12-21T03:53:43.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.979+0000 7fcd66ffd700 1 --2- 172.21.15.78:0/473925840 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcd6005c2a0 0x7fcd6005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:43.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.979+0000 7fcd66ffd700 1 -- 172.21.15.78:0/473925840 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fcd68077730 con 0x7fcd780f67d0 2023-12-21T03:53:43.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.980+0000 7fcd75d9b700 1 --2- 172.21.15.78:0/473925840 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcd6005c2a0 0x7fcd6005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:43.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.980+0000 7fcd75d9b700 1 --2- 172.21.15.78:0/473925840 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcd6005c2a0 0x7fcd6005e760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fcd6c004c40 tx=0x7fcd6c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:43.983+0000 7fcd66ffd700 1 -- 172.21.15.78:0/473925840 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd6804a7b0 con 0x7fcd780f67d0 2023-12-21T03:53:44.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:41.656591+0000 mgr.smithi078.zidbgq (mgr.14162) 474 : cluster [DBG] pgmap v451: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:44.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.191+0000 7fcd7ce99700 1 -- 172.21.15.78:0/473925840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcd78027a00 con 0x7fcd780f67d0 2023-12-21T03:53:44.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.192+0000 7fcd66ffd700 1 -- 172.21.15.78:0/473925840 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fcd6804ddd0 con 0x7fcd780f67d0 2023-12-21T03:53:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:44.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.193+0000 7fcd7ce99700 1 -- 172.21.15.78:0/473925840 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcd6005c2a0 msgr2=0x7fcd6005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:44.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.193+0000 7fcd7ce99700 1 --2- 172.21.15.78:0/473925840 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcd6005c2a0 0x7fcd6005e760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fcd6c004c40 tx=0x7fcd6c00b040).stop 2023-12-21T03:53:44.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.193+0000 7fcd7ce99700 1 -- 172.21.15.78:0/473925840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780f67d0 msgr2=0x7fcd781070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:44.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.193+0000 7fcd7ce99700 1 --2- 172.21.15.78:0/473925840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780f67d0 0x7fcd781070b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fcd68004e30 tx=0x7fcd68009b30).stop 2023-12-21T03:53:44.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.193+0000 7fcd7ce99700 1 -- 172.21.15.78:0/473925840 shutdown_connections 2023-12-21T03:53:44.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.193+0000 7fcd7ce99700 1 --2- 172.21.15.78:0/473925840 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcd6005c2a0 0x7fcd6005e760 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:44.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.193+0000 7fcd7ce99700 1 --2- 172.21.15.78:0/473925840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780f67d0 0x7fcd781070b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:44.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.193+0000 7fcd7ce99700 1 --2- 172.21.15.78:0/473925840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd780f7a70 0x7fcd781075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:44.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.193+0000 7fcd7ce99700 1 -- 172.21.15.78:0/473925840 >> 172.21.15.78:0/473925840 conn(0x7fcd780f1f50 msgr2=0x7fcd780fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:44.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.193+0000 7fcd7ce99700 1 -- 172.21.15.78:0/473925840 shutdown_connections 2023-12-21T03:53:44.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:44.193+0000 7fcd7ce99700 1 -- 172.21.15.78:0/473925840 wait complete. 2023-12-21T03:53:44.588 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:44.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:44 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:44.193008+0000 mon.smithi078 (mon.0) 515 : audit [DBG] from='client.? 172.21.15.78:0/473925840' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:45.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:44 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:44.193008+0000 mon.smithi078 (mon.0) 515 : audit [DBG] from='client.? 172.21.15.78:0/473925840' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:43.657018+0000 mgr.smithi078.zidbgq (mgr.14162) 475 : cluster [DBG] pgmap v452: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:46.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:43.657018+0000 mgr.smithi078.zidbgq (mgr.14162) 475 : cluster [DBG] pgmap v452: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:47.589 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:45.657255+0000 mgr.smithi078.zidbgq (mgr.14162) 476 : cluster [DBG] pgmap v453: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:47.828 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:48.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:45.657255+0000 mgr.smithi078.zidbgq (mgr.14162) 476 : cluster [DBG] pgmap v453: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.869+0000 7fc729f16700 1 -- 172.21.15.78:0/686392288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72406a3b0 msgr2=0x7fc72406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.869+0000 7fc729f16700 1 --2- 172.21.15.78:0/686392288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72406a3b0 0x7fc72406a830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fc710004660 tx=0x7fc710009b30).stop 2023-12-21T03:53:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.870+0000 7fc729f16700 1 -- 172.21.15.78:0/686392288 shutdown_connections 2023-12-21T03:53:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.870+0000 7fc729f16700 1 --2- 172.21.15.78:0/686392288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72406a3b0 0x7fc72406a830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.870+0000 7fc729f16700 1 --2- 172.21.15.78:0/686392288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc72406b9c0 0x7fc724069ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.870+0000 7fc729f16700 1 -- 172.21.15.78:0/686392288 >> 172.21.15.78:0/686392288 conn(0x7fc7240f1d70 msgr2=0x7fc7240f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.870+0000 7fc729f16700 1 -- 172.21.15.78:0/686392288 shutdown_connections 2023-12-21T03:53:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.870+0000 7fc729f16700 1 -- 172.21.15.78:0/686392288 wait complete. 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.870+0000 7fc729f16700 1 Processor -- start 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.870+0000 7fc729f16700 1 -- start start 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.870+0000 7fc729f16700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72406a3b0 0x7fc724109080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc729f16700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc72406b9c0 0x7fc724109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc729f16700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc724109b20 con 0x7fc72406a3b0 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc729f16700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc724109c60 con 0x7fc72406b9c0 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc728f14700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72406a3b0 0x7fc724109080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc728f14700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72406a3b0 0x7fc724109080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44750/0 (socket says 172.21.15.78:44750) 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc728f14700 1 -- 172.21.15.78:0/2432677948 learned_addr learned my addr 172.21.15.78:0/2432677948 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc723fff700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc72406b9c0 0x7fc724109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc728f14700 1 -- 172.21.15.78:0/2432677948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc72406b9c0 msgr2=0x7fc724109570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc728f14700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc72406b9c0 0x7fc724109570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc728f14700 1 -- 172.21.15.78:0/2432677948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc710005040 con 0x7fc72406a3b0 2023-12-21T03:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc723fff700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc72406b9c0 0x7fc724109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc728f14700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72406a3b0 0x7fc724109080 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fc7180026d0 tx=0x7fc718009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc7217fa700 1 -- 172.21.15.78:0/2432677948 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc718009320 con 0x7fc72406a3b0 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc7217fa700 1 -- 172.21.15.78:0/2432677948 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc71800f040 con 0x7fc72406a3b0 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc729f16700 1 -- 172.21.15.78:0/2432677948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72410a600 con 0x7fc72406a3b0 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc7217fa700 1 -- 172.21.15.78:0/2432677948 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7180147e0 con 0x7fc72406a3b0 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.871+0000 7fc729f16700 1 -- 172.21.15.78:0/2432677948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72410abc0 con 0x7fc72406a3b0 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.872+0000 7fc729f16700 1 -- 172.21.15.78:0/2432677948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc724045ad0 con 0x7fc72406a3b0 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.872+0000 7fc7217fa700 1 -- 172.21.15.78:0/2432677948 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc718009480 con 0x7fc72406a3b0 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.873+0000 7fc7217fa700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc71405c2b0 0x7fc71405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.873+0000 7fc7217fa700 1 -- 172.21.15.78:0/2432677948 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc718076270 con 0x7fc72406a3b0 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.873+0000 7fc723fff700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc71405c2b0 0x7fc71405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.873+0000 7fc723fff700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc71405c2b0 0x7fc71405e770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fc710005010 tx=0x7fc710009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:48.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:48.877+0000 7fc7217fa700 1 -- 172.21.15.78:0/2432677948 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc718049270 con 0x7fc72406a3b0 2023-12-21T03:53:49.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.087+0000 7fc729f16700 1 -- 172.21.15.78:0/2432677948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc724101110 con 0x7fc72406a3b0 2023-12-21T03:53:49.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.087+0000 7fc7217fa700 1 -- 172.21.15.78:0/2432677948 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fc71804c890 con 0x7fc72406a3b0 2023-12-21T03:53:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:49.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.088+0000 7fc729f16700 1 -- 172.21.15.78:0/2432677948 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc71405c2b0 msgr2=0x7fc71405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:49.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.088+0000 7fc729f16700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc71405c2b0 0x7fc71405e770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fc710005010 tx=0x7fc710009b30).stop 2023-12-21T03:53:49.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.088+0000 7fc729f16700 1 -- 172.21.15.78:0/2432677948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72406a3b0 msgr2=0x7fc724109080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:49.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.088+0000 7fc729f16700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72406a3b0 0x7fc724109080 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fc7180026d0 tx=0x7fc718009de0).stop 2023-12-21T03:53:49.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.089+0000 7fc729f16700 1 -- 172.21.15.78:0/2432677948 shutdown_connections 2023-12-21T03:53:49.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.089+0000 7fc729f16700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc71405c2b0 0x7fc71405e770 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:49.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.089+0000 7fc729f16700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72406a3b0 0x7fc724109080 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:49.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.089+0000 7fc729f16700 1 --2- 172.21.15.78:0/2432677948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc72406b9c0 0x7fc724109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:49.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.089+0000 7fc729f16700 1 -- 172.21.15.78:0/2432677948 >> 172.21.15.78:0/2432677948 conn(0x7fc7240f1d70 msgr2=0x7fc7240fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:49.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.089+0000 7fc729f16700 1 -- 172.21.15.78:0/2432677948 shutdown_connections 2023-12-21T03:53:49.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:49.089+0000 7fc729f16700 1 -- 172.21.15.78:0/2432677948 wait complete. 2023-12-21T03:53:49.462 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:49.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:47.657572+0000 mgr.smithi078.zidbgq (mgr.14162) 477 : cluster [DBG] pgmap v454: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:49.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:49.088444+0000 mon.smithi078 (mon.0) 516 : audit [DBG] from='client.? 172.21.15.78:0/2432677948' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:50.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:47.657572+0000 mgr.smithi078.zidbgq (mgr.14162) 477 : cluster [DBG] pgmap v454: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:50.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:49.088444+0000 mon.smithi078 (mon.0) 516 : audit [DBG] from='client.? 172.21.15.78:0/2432677948' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:49.658124+0000 mgr.smithi078.zidbgq (mgr.14162) 478 : cluster [DBG] pgmap v455: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:52.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:49.658124+0000 mgr.smithi078.zidbgq (mgr.14162) 478 : cluster [DBG] pgmap v455: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:52.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:52.701 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.658+0000 7ff01f59e700 1 -- 172.21.15.78:0/1475272224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0200f67b0 msgr2=0x7ff0200f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.658+0000 7ff01f59e700 1 --2- 172.21.15.78:0/1475272224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0200f67b0 0x7ff0200f6c10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7ff014004660 tx=0x7ff014009b30).stop 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.658+0000 7ff01f59e700 1 -- 172.21.15.78:0/1475272224 shutdown_connections 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.658+0000 7ff01f59e700 1 --2- 172.21.15.78:0/1475272224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0200f67b0 0x7ff0200f6c10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.658+0000 7ff01f59e700 1 --2- 172.21.15.78:0/1475272224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0200f5510 0x7ff0200f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.658+0000 7ff01f59e700 1 -- 172.21.15.78:0/1475272224 >> 172.21.15.78:0/1475272224 conn(0x7ff0200f0c70 msgr2=0x7ff0200f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01f59e700 1 -- 172.21.15.78:0/1475272224 shutdown_connections 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01f59e700 1 -- 172.21.15.78:0/1475272224 wait complete. 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01f59e700 1 Processor -- start 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01f59e700 1 -- start start 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01f59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0200f5510 0x7ff020109bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01f59e700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0200f67b0 0x7ff02010c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01f59e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff020060920 con 0x7ff0200f5510 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01f59e700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff020060a60 con 0x7ff0200f67b0 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0200f5510 0x7ff020109bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0200f5510 0x7ff020109bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44768/0 (socket says 172.21.15.78:44768) 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01e59c700 1 -- 172.21.15.78:0/3880123704 learned_addr learned my addr 172.21.15.78:0/3880123704 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.659+0000 7ff01dd9b700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0200f67b0 0x7ff02010c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.660+0000 7ff01e59c700 1 -- 172.21.15.78:0/3880123704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0200f67b0 msgr2=0x7ff02010c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.660+0000 7ff01e59c700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0200f67b0 0x7ff02010c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.660+0000 7ff01e59c700 1 -- 172.21.15.78:0/3880123704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff014005040 con 0x7ff0200f5510 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.660+0000 7ff01dd9b700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0200f67b0 0x7ff02010c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.660+0000 7ff01e59c700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0200f5510 0x7ff020109bf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7ff01000c530 tx=0x7ff010007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.660+0000 7ff00effd700 1 -- 172.21.15.78:0/3880123704 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff01000f650 con 0x7ff0200f5510 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.660+0000 7ff01f59e700 1 -- 172.21.15.78:0/3880123704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff02010c6a0 con 0x7ff0200f5510 2023-12-21T03:53:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.660+0000 7ff00effd700 1 -- 172.21.15.78:0/3880123704 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff01000fc60 con 0x7ff0200f5510 2023-12-21T03:53:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.660+0000 7ff00effd700 1 -- 172.21.15.78:0/3880123704 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff010012d20 con 0x7ff0200f5510 2023-12-21T03:53:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.660+0000 7ff01f59e700 1 -- 172.21.15.78:0/3880123704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff02010cc30 con 0x7ff0200f5510 2023-12-21T03:53:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.661+0000 7ff01f59e700 1 -- 172.21.15.78:0/3880123704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff020059100 con 0x7ff0200f5510 2023-12-21T03:53:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.661+0000 7ff00effd700 1 -- 172.21.15.78:0/3880123704 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff010019030 con 0x7ff0200f5510 2023-12-21T03:53:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.661+0000 7ff00effd700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff00805c2b0 0x7ff00805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.661+0000 7ff00effd700 1 -- 172.21.15.78:0/3880123704 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff010075830 con 0x7ff0200f5510 2023-12-21T03:53:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.662+0000 7ff01dd9b700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff00805c2b0 0x7ff00805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.662+0000 7ff01dd9b700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff00805c2b0 0x7ff00805e770 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ff014009d60 tx=0x7ff014009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.665+0000 7ff00effd700 1 -- 172.21.15.78:0/3880123704 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff010048880 con 0x7ff0200f5510 2023-12-21T03:53:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:51.658458+0000 mgr.smithi078.zidbgq (mgr.14162) 479 : cluster [DBG] pgmap v456: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:53.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.871+0000 7ff01f59e700 1 -- 172.21.15.78:0/3880123704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff020045ad0 con 0x7ff0200f5510 2023-12-21T03:53:53.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.872+0000 7ff00effd700 1 -- 172.21.15.78:0/3880123704 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7ff010017070 con 0x7ff0200f5510 2023-12-21T03:53:53.873 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.873+0000 7ff01f59e700 1 -- 172.21.15.78:0/3880123704 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff00805c2b0 msgr2=0x7ff00805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.873+0000 7ff01f59e700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff00805c2b0 0x7ff00805e770 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ff014009d60 tx=0x7ff014009b30).stop 2023-12-21T03:53:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.873+0000 7ff01f59e700 1 -- 172.21.15.78:0/3880123704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0200f5510 msgr2=0x7ff020109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.873+0000 7ff01f59e700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0200f5510 0x7ff020109bf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7ff01000c530 tx=0x7ff010007e10).stop 2023-12-21T03:53:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.873+0000 7ff01f59e700 1 -- 172.21.15.78:0/3880123704 shutdown_connections 2023-12-21T03:53:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.873+0000 7ff01f59e700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7ff00805c2b0 0x7ff00805e770 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.873+0000 7ff01f59e700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0200f5510 0x7ff020109bf0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:53.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.873+0000 7ff01f59e700 1 --2- 172.21.15.78:0/3880123704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0200f67b0 0x7ff02010c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:53.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.873+0000 7ff01f59e700 1 -- 172.21.15.78:0/3880123704 >> 172.21.15.78:0/3880123704 conn(0x7ff0200f0c70 msgr2=0x7ff0200fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:53.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.874+0000 7ff01f59e700 1 -- 172.21.15.78:0/3880123704 shutdown_connections 2023-12-21T03:53:53.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:53.874+0000 7ff01f59e700 1 -- 172.21.15.78:0/3880123704 wait complete. 2023-12-21T03:53:54.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:51.658458+0000 mgr.smithi078.zidbgq (mgr.14162) 479 : cluster [DBG] pgmap v456: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:54.244 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:53:55.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:54 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:53.873190+0000 mon.smithi078 (mon.0) 517 : audit [DBG] from='client.? 172.21.15.78:0/3880123704' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:55.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:54 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:53.873190+0000 mon.smithi078 (mon.0) 517 : audit [DBG] from='client.? 172.21.15.78:0/3880123704' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:53:56.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:53.658737+0000 mgr.smithi078.zidbgq (mgr.14162) 480 : cluster [DBG] pgmap v457: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:56.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:53.658737+0000 mgr.smithi078.zidbgq (mgr.14162) 480 : cluster [DBG] pgmap v457: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:57.245 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:53:57.486 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:53:57.948 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:55.659083+0000 mgr.smithi078.zidbgq (mgr.14162) 481 : cluster [DBG] pgmap v458: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:58.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:55.659083+0000 mgr.smithi078.zidbgq (mgr.14162) 481 : cluster [DBG] pgmap v458: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:53:58.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.444+0000 7f018a6c3700 1 -- 172.21.15.78:0/1172236295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01840f67d0 msgr2=0x7f01840f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:58.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.444+0000 7f018a6c3700 1 --2- 172.21.15.78:0/1172236295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01840f67d0 0x7f01840f6bf0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f0174004660 tx=0x7f0174009b30).stop 2023-12-21T03:53:58.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.444+0000 7f018a6c3700 1 -- 172.21.15.78:0/1172236295 shutdown_connections 2023-12-21T03:53:58.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.444+0000 7f018a6c3700 1 --2- 172.21.15.78:0/1172236295 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01840f7a70 0x7f01840f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:58.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.444+0000 7f018a6c3700 1 --2- 172.21.15.78:0/1172236295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01840f67d0 0x7f01840f6bf0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:58.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.444+0000 7f018a6c3700 1 -- 172.21.15.78:0/1172236295 >> 172.21.15.78:0/1172236295 conn(0x7f01840f1f50 msgr2=0x7f01840f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:58.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.444+0000 7f018a6c3700 1 -- 172.21.15.78:0/1172236295 shutdown_connections 2023-12-21T03:53:58.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.444+0000 7f018a6c3700 1 -- 172.21.15.78:0/1172236295 wait complete. 2023-12-21T03:53:58.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f018a6c3700 1 Processor -- start 2023-12-21T03:53:58.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f018a6c3700 1 -- start start 2023-12-21T03:53:58.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f018a6c3700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01840f67d0 0x7f018406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f018a6c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01840f7a70 0x7f0184069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f018a6c3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018406a380 con 0x7f01840f7a70 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f018a6c3700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018406a4c0 con 0x7f01840f67d0 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f01837fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01840f7a70 0x7f0184069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f01837fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01840f7a70 0x7f0184069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58470/0 (socket says 172.21.15.78:58470) 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f01837fe700 1 -- 172.21.15.78:0/1324099118 learned_addr learned my addr 172.21.15.78:0/1324099118 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f0183fff700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01840f67d0 0x7f018406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f01837fe700 1 -- 172.21.15.78:0/1324099118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01840f67d0 msgr2=0x7f018406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f01837fe700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01840f67d0 0x7f018406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f01837fe700 1 -- 172.21.15.78:0/1324099118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0174005040 con 0x7f01840f7a70 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.445+0000 7f0183fff700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01840f67d0 0x7f018406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:53:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.446+0000 7f01837fe700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01840f7a70 0x7f0184069e20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f017800c530 tx=0x7f0178007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.446+0000 7f0180ff9700 1 -- 172.21.15.78:0/1324099118 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f017800f650 con 0x7f01840f7a70 2023-12-21T03:53:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.446+0000 7f0180ff9700 1 -- 172.21.15.78:0/1324099118 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f017800fc60 con 0x7f01840f7a70 2023-12-21T03:53:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.446+0000 7f018a6c3700 1 -- 172.21.15.78:0/1324099118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018406a7a0 con 0x7f01840f7a70 2023-12-21T03:53:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.446+0000 7f0180ff9700 1 -- 172.21.15.78:0/1324099118 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0178012c80 con 0x7f01840f7a70 2023-12-21T03:53:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.446+0000 7f018a6c3700 1 -- 172.21.15.78:0/1324099118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018410f0e0 con 0x7f01840f7a70 2023-12-21T03:53:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.447+0000 7f018a6c3700 1 -- 172.21.15.78:0/1324099118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0184045ad0 con 0x7f01840f7a70 2023-12-21T03:53:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.447+0000 7f0180ff9700 1 -- 172.21.15.78:0/1324099118 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0178010070 con 0x7f01840f7a70 2023-12-21T03:53:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.447+0000 7f0180ff9700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f016c05c2a0 0x7f016c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:53:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.447+0000 7f0180ff9700 1 -- 172.21.15.78:0/1324099118 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f01780757e0 con 0x7f01840f7a70 2023-12-21T03:53:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.447+0000 7f0183fff700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f016c05c2a0 0x7f016c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:53:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.448+0000 7f0183fff700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f016c05c2a0 0x7f016c05e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f018406b730 tx=0x7f0174009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:53:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.451+0000 7f0180ff9700 1 -- 172.21.15.78:0/1324099118 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01780452b0 con 0x7f01840f7a70 2023-12-21T03:53:58.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.660+0000 7f018a6c3700 1 -- 172.21.15.78:0/1324099118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0184027a00 con 0x7f01840f7a70 2023-12-21T03:53:58.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.660+0000 7f0180ff9700 1 -- 172.21.15.78:0/1324099118 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f0178048860 con 0x7f01840f7a70 2023-12-21T03:53:58.662 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:53:58.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.662+0000 7f018a6c3700 1 -- 172.21.15.78:0/1324099118 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f016c05c2a0 msgr2=0x7f016c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:58.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.662+0000 7f018a6c3700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f016c05c2a0 0x7f016c05e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f018406b730 tx=0x7f0174009b30).stop 2023-12-21T03:53:58.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.662+0000 7f018a6c3700 1 -- 172.21.15.78:0/1324099118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01840f7a70 msgr2=0x7f0184069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:53:58.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.662+0000 7f018a6c3700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01840f7a70 0x7f0184069e20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f017800c530 tx=0x7f0178007e20).stop 2023-12-21T03:53:58.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.662+0000 7f018a6c3700 1 -- 172.21.15.78:0/1324099118 shutdown_connections 2023-12-21T03:53:58.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.662+0000 7f018a6c3700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f016c05c2a0 0x7f016c05e760 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:58.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.662+0000 7f018a6c3700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01840f67d0 0x7f018406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:58.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.662+0000 7f018a6c3700 1 --2- 172.21.15.78:0/1324099118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01840f7a70 0x7f0184069e20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:53:58.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.662+0000 7f018a6c3700 1 -- 172.21.15.78:0/1324099118 >> 172.21.15.78:0/1324099118 conn(0x7f01840f1f50 msgr2=0x7f01840fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:53:58.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.662+0000 7f018a6c3700 1 -- 172.21.15.78:0/1324099118 shutdown_connections 2023-12-21T03:53:58.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:53:58.662+0000 7f018a6c3700 1 -- 172.21.15.78:0/1324099118 wait complete. 2023-12-21T03:53:59.026 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:00.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:57.659402+0000 mgr.smithi078.zidbgq (mgr.14162) 482 : cluster [DBG] pgmap v459: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:00.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:53:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:53:58.661725+0000 mon.smithi078 (mon.0) 518 : audit [DBG] from='client.? 172.21.15.78:0/1324099118' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:00.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:57.659402+0000 mgr.smithi078.zidbgq (mgr.14162) 482 : cluster [DBG] pgmap v459: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:00.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:53:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:53:58.661725+0000 mon.smithi078 (mon.0) 518 : audit [DBG] from='client.? 172.21.15.78:0/1324099118' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:00.835 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:00.356336+0000 mon.smithi078 (mon.0) 519 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi148", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-21T03:54:00.835 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:00.357253+0000 mon.smithi078 (mon.0) 520 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:54:01.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:00.356336+0000 mon.smithi078 (mon.0) 519 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi148", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-21T03:54:01.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:00.357253+0000 mon.smithi078 (mon.0) 520 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:54:01.968 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:53:59.659798+0000 mgr.smithi078.zidbgq (mgr.14162) 483 : cluster [DBG] pgmap v460: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:54:00.355588+0000 mgr.smithi078.zidbgq (mgr.14162) 484 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi078 ... 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Non-zero exit code 1 from systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: main() 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: r = ctx.func(ctx) 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: return func(ctx) 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:54:01.969 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: ports=daemon_ports) 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: get_unit_name(fsid, daemon_type, daemon_id)]) 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: RuntimeError: Failed command: systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: yield (conn, connr) 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: code, '\n'.join(err))) 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi078 ... 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Non-zero exit code 1 from systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2023-12-21T03:54:01.970 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: main() 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: r = ctx.func(ctx) 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: return func(ctx) 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: ports=daemon_ports) 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: get_unit_name(fsid, daemon_type, daemon_id)]) 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: RuntimeError: Failed command: systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:54:01.971 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:54:00.355934+0000 mgr.smithi078.zidbgq (mgr.14162) 485 : cephadm [INF] Reconfiguring crash.smithi148 (monmap changed)... 2023-12-21T03:54:01.972 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:54:00.357981+0000 mgr.smithi078.zidbgq (mgr.14162) 486 : cephadm [INF] Reconfiguring daemon crash.smithi148 on smithi148 2023-12-21T03:54:01.972 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:01.237882+0000 mon.smithi078 (mon.0) 521 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:54:01.972 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:01.238859+0000 mon.smithi078 (mon.0) 522 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi148.tgpyxd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-21T03:54:01.972 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:01.239754+0000 mon.smithi078 (mon.0) 523 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-21T03:54:01.972 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:01.240432+0000 mon.smithi078 (mon.0) 524 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:54:02.027 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:02.062 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:53:59.659798+0000 mgr.smithi078.zidbgq (mgr.14162) 483 : cluster [DBG] pgmap v460: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:02.062 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:54:00.355588+0000 mgr.smithi078.zidbgq (mgr.14162) 484 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi078 ... 2023-12-21T03:54:02.063 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 1 from systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:54:02.063 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2023-12-21T03:54:02.063 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2023-12-21T03:54:02.063 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:54:02.063 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:54:02.063 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ports=daemon_ports) 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: get_unit_name(fsid, daemon_type, daemon_id)]) 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: Failed command: systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: yield (conn, connr) 2023-12-21T03:54:02.064 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: code, '\n'.join(err))) 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi078 ... 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 1 from systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:54:02.065 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ports=daemon_ports) 2023-12-21T03:54:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2023-12-21T03:54:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: get_unit_name(fsid, daemon_type, daemon_id)]) 2023-12-21T03:54:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:54:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:54:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: Failed command: systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi078 2023-12-21T03:54:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:54:00.355934+0000 mgr.smithi078.zidbgq (mgr.14162) 485 : cephadm [INF] Reconfiguring crash.smithi148 (monmap changed)... 2023-12-21T03:54:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:54:00.357981+0000 mgr.smithi078.zidbgq (mgr.14162) 486 : cephadm [INF] Reconfiguring daemon crash.smithi148 on smithi148 2023-12-21T03:54:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:01.237882+0000 mon.smithi078 (mon.0) 521 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:54:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:01.238859+0000 mon.smithi078 (mon.0) 522 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi148.tgpyxd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-21T03:54:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:01.239754+0000 mon.smithi078 (mon.0) 523 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-21T03:54:02.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:01.240432+0000 mon.smithi078 (mon.0) 524 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:54:02.269 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:02.714 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:54:01.238515+0000 mgr.smithi078.zidbgq (mgr.14162) 487 : cephadm [INF] Reconfiguring mgr.smithi148.tgpyxd (monmap changed)... 2023-12-21T03:54:02.714 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:54:01.241063+0000 mgr.smithi078.zidbgq (mgr.14162) 488 : cephadm [INF] Reconfiguring daemon mgr.smithi148.tgpyxd on smithi148 2023-12-21T03:54:02.714 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:02.168936+0000 mon.smithi078 (mon.0) 525 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:54:02.714 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:02.169767+0000 mon.smithi078 (mon.0) 526 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-21T03:54:02.715 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:02.170316+0000 mon.smithi078 (mon.0) 527 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-21T03:54:02.715 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:02.170915+0000 mon.smithi078 (mon.0) 528 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:54:02.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:02 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:54:01.238515+0000 mgr.smithi078.zidbgq (mgr.14162) 487 : cephadm [INF] Reconfiguring mgr.smithi148.tgpyxd (monmap changed)... 2023-12-21T03:54:02.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:02 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:54:01.241063+0000 mgr.smithi078.zidbgq (mgr.14162) 488 : cephadm [INF] Reconfiguring daemon mgr.smithi148.tgpyxd on smithi148 2023-12-21T03:54:02.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:02 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:02.168936+0000 mon.smithi078 (mon.0) 525 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:54:02.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:02 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:02.169767+0000 mon.smithi078 (mon.0) 526 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-21T03:54:02.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:02 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:02.170316+0000 mon.smithi078 (mon.0) 527 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-21T03:54:02.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:02 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:02.170915+0000 mon.smithi078 (mon.0) 528 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-21T03:54:03.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.216+0000 7fc72877b700 1 -- 172.21.15.78:0/903696229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7200f9920 msgr2=0x7fc7200fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:03.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.216+0000 7fc72877b700 1 --2- 172.21.15.78:0/903696229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7200f9920 0x7fc7200fbd70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fc710004660 tx=0x7fc710009b30).stop 2023-12-21T03:54:03.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.216+0000 7fc72877b700 1 -- 172.21.15.78:0/903696229 shutdown_connections 2023-12-21T03:54:03.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.216+0000 7fc72877b700 1 --2- 172.21.15.78:0/903696229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7200f9920 0x7fc7200fbd70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.216+0000 7fc72877b700 1 --2- 172.21.15.78:0/903696229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720060870 0x7fc7200f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.216+0000 7fc72877b700 1 -- 172.21.15.78:0/903696229 >> 172.21.15.78:0/903696229 conn(0x7fc7200f0e20 msgr2=0x7fc7200f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.216+0000 7fc72877b700 1 -- 172.21.15.78:0/903696229 shutdown_connections 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.216+0000 7fc72877b700 1 -- 172.21.15.78:0/903696229 wait complete. 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc72877b700 1 Processor -- start 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc72877b700 1 -- start start 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc72877b700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc720060870 0x7fc720106ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc72877b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7200f9920 0x7fc7201074e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc72877b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc720107a90 con 0x7fc7200f9920 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc72877b700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc720107bd0 con 0x7fc720060870 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc725d16700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7200f9920 0x7fc7201074e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc725d16700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7200f9920 0x7fc7201074e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58494/0 (socket says 172.21.15.78:58494) 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc725d16700 1 -- 172.21.15.78:0/4178792076 learned_addr learned my addr 172.21.15.78:0/4178792076 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:03.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc726517700 1 --2- 172.21.15.78:0/4178792076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc720060870 0x7fc720106ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:03.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc725d16700 1 -- 172.21.15.78:0/4178792076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc720060870 msgr2=0x7fc720106ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:03.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc725d16700 1 --2- 172.21.15.78:0/4178792076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc720060870 0x7fc720106ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:03.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.217+0000 7fc725d16700 1 -- 172.21.15.78:0/4178792076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc710005040 con 0x7fc7200f9920 2023-12-21T03:54:03.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.218+0000 7fc725d16700 1 --2- 172.21.15.78:0/4178792076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7200f9920 0x7fc7201074e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fc710000c00 tx=0x7fc71000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:03.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.218+0000 7fc716ffd700 1 -- 172.21.15.78:0/4178792076 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc710019070 con 0x7fc7200f9920 2023-12-21T03:54:03.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.218+0000 7fc72877b700 1 -- 172.21.15.78:0/4178792076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc720108510 con 0x7fc7200f9920 2023-12-21T03:54:03.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.218+0000 7fc716ffd700 1 -- 172.21.15.78:0/4178792076 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc710007e70 con 0x7fc7200f9920 2023-12-21T03:54:03.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.218+0000 7fc716ffd700 1 -- 172.21.15.78:0/4178792076 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc710014d40 con 0x7fc7200f9920 2023-12-21T03:54:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.218+0000 7fc72877b700 1 -- 172.21.15.78:0/4178792076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc720108a70 con 0x7fc7200f9920 2023-12-21T03:54:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.219+0000 7fc72877b700 1 -- 172.21.15.78:0/4178792076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc720045ad0 con 0x7fc7200f9920 2023-12-21T03:54:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.219+0000 7fc716ffd700 1 -- 172.21.15.78:0/4178792076 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc710025070 con 0x7fc7200f9920 2023-12-21T03:54:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.219+0000 7fc716ffd700 1 --2- 172.21.15.78:0/4178792076 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc70c05c2a0 0x7fc70c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.219+0000 7fc716ffd700 1 -- 172.21.15.78:0/4178792076 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc710076b10 con 0x7fc7200f9920 2023-12-21T03:54:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.219+0000 7fc726517700 1 --2- 172.21.15.78:0/4178792076 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc70c05c2a0 0x7fc70c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.220+0000 7fc726517700 1 --2- 172.21.15.78:0/4178792076 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc70c05c2a0 0x7fc70c05e760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fc71c009fd0 tx=0x7fc71c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:03.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.223+0000 7fc716ffd700 1 -- 172.21.15.78:0/4178792076 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc710049b10 con 0x7fc7200f9920 2023-12-21T03:54:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.430+0000 7fc72877b700 1 -- 172.21.15.78:0/4178792076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc720027a00 con 0x7fc7200f9920 2023-12-21T03:54:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.430+0000 7fc716ffd700 1 -- 172.21.15.78:0/4178792076 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fc71004d130 con 0x7fc7200f9920 2023-12-21T03:54:03.431 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:03.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.432+0000 7fc72877b700 1 -- 172.21.15.78:0/4178792076 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc70c05c2a0 msgr2=0x7fc70c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:03.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.432+0000 7fc72877b700 1 --2- 172.21.15.78:0/4178792076 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc70c05c2a0 0x7fc70c05e760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fc71c009fd0 tx=0x7fc71c00b040).stop 2023-12-21T03:54:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.432+0000 7fc72877b700 1 -- 172.21.15.78:0/4178792076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7200f9920 msgr2=0x7fc7201074e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.432+0000 7fc72877b700 1 --2- 172.21.15.78:0/4178792076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7200f9920 0x7fc7201074e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fc710000c00 tx=0x7fc71000b040).stop 2023-12-21T03:54:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.432+0000 7fc72877b700 1 -- 172.21.15.78:0/4178792076 shutdown_connections 2023-12-21T03:54:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.432+0000 7fc72877b700 1 --2- 172.21.15.78:0/4178792076 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fc70c05c2a0 0x7fc70c05e760 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.432+0000 7fc72877b700 1 --2- 172.21.15.78:0/4178792076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc720060870 0x7fc720106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.432+0000 7fc72877b700 1 --2- 172.21.15.78:0/4178792076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7200f9920 0x7fc7201074e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.432+0000 7fc72877b700 1 -- 172.21.15.78:0/4178792076 >> 172.21.15.78:0/4178792076 conn(0x7fc7200f0e20 msgr2=0x7fc7200f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.432+0000 7fc72877b700 1 -- 172.21.15.78:0/4178792076 shutdown_connections 2023-12-21T03:54:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:03.432+0000 7fc72877b700 1 -- 172.21.15.78:0/4178792076 wait complete. 2023-12-21T03:54:03.726 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:01.660297+0000 mgr.smithi078.zidbgq (mgr.14162) 489 : cluster [DBG] pgmap v461: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:03.726 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:54:02.169554+0000 mgr.smithi078.zidbgq (mgr.14162) 490 : cephadm [INF] Reconfiguring mon.smithi148 (monmap changed)... 2023-12-21T03:54:03.726 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:54:02.171443+0000 mgr.smithi078.zidbgq (mgr.14162) 491 : cephadm [INF] Reconfiguring daemon mon.smithi148 on smithi148 2023-12-21T03:54:03.726 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:03.002012+0000 mon.smithi078 (mon.0) 529 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:54:03.726 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:54:03.002476+0000 mgr.smithi078.zidbgq (mgr.14162) 492 : cephadm [INF] Reconfiguring node-exporter.smithi148 (unknown last config time)... 2023-12-21T03:54:03.726 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:54:03.002582+0000 mgr.smithi078.zidbgq (mgr.14162) 493 : cephadm [INF] Reconfiguring daemon node-exporter.smithi148 on smithi148 2023-12-21T03:54:03.726 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:03.431637+0000 mon.smithi078 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.78:0/4178792076' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:03.809 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:04.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:01.660297+0000 mgr.smithi078.zidbgq (mgr.14162) 489 : cluster [DBG] pgmap v461: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:04.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:54:02.169554+0000 mgr.smithi078.zidbgq (mgr.14162) 490 : cephadm [INF] Reconfiguring mon.smithi148 (monmap changed)... 2023-12-21T03:54:04.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:54:02.171443+0000 mgr.smithi078.zidbgq (mgr.14162) 491 : cephadm [INF] Reconfiguring daemon mon.smithi148 on smithi148 2023-12-21T03:54:04.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:03.002012+0000 mon.smithi078 (mon.0) 529 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:54:04.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:54:03.002476+0000 mgr.smithi078.zidbgq (mgr.14162) 492 : cephadm [INF] Reconfiguring node-exporter.smithi148 (unknown last config time)... 2023-12-21T03:54:04.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:54:03.002582+0000 mgr.smithi078.zidbgq (mgr.14162) 493 : cephadm [INF] Reconfiguring daemon node-exporter.smithi148 on smithi148 2023-12-21T03:54:04.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:03.431637+0000 mon.smithi078 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.78:0/4178792076' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:03.660545+0000 mgr.smithi078.zidbgq (mgr.14162) 494 : cluster [DBG] pgmap v462: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:05.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:03.660545+0000 mgr.smithi078.zidbgq (mgr.14162) 494 : cluster [DBG] pgmap v462: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:06.811 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:07.049 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:07.522 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:05.660828+0000 mgr.smithi078.zidbgq (mgr.14162) 495 : cluster [DBG] pgmap v463: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:07.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:05.660828+0000 mgr.smithi078.zidbgq (mgr.14162) 495 : cluster [DBG] pgmap v463: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:08.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.005+0000 7fce11e06700 1 -- 172.21.15.78:0/1753034765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce0c060870 msgr2=0x7fce0c0f93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:08.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.005+0000 7fce11e06700 1 --2- 172.21.15.78:0/1753034765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce0c060870 0x7fce0c0f93c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fcdfc004660 tx=0x7fcdfc009b30).stop 2023-12-21T03:54:08.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.005+0000 7fce11e06700 1 -- 172.21.15.78:0/1753034765 shutdown_connections 2023-12-21T03:54:08.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.005+0000 7fce11e06700 1 --2- 172.21.15.78:0/1753034765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce0c0f9940 0x7fce0c0fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:54:08.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.005+0000 7fce11e06700 1 --2- 172.21.15.78:0/1753034765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce0c060870 0x7fce0c0f93c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:08.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.005+0000 7fce11e06700 1 -- 172.21.15.78:0/1753034765 >> 172.21.15.78:0/1753034765 conn(0x7fce0c0f0e20 msgr2=0x7fce0c0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.005+0000 7fce11e06700 1 -- 172.21.15.78:0/1753034765 shutdown_connections 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.005+0000 7fce11e06700 1 -- 172.21.15.78:0/1753034765 wait complete. 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce11e06700 1 Processor -- start 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce11e06700 1 -- start start 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce11e06700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce0c060870 0x7fce0c1070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce11e06700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce0c0f9940 0x7fce0c1075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce11e06700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c107bf0 con 0x7fce0c060870 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce11e06700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c1084a0 con 0x7fce0c0f9940 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce0b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce0c060870 0x7fce0c1070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce0b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce0c060870 0x7fce0c1070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58520/0 (socket says 172.21.15.78:58520) 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce0b7fe700 1 -- 172.21.15.78:0/295670065 learned_addr learned my addr 172.21.15.78:0/295670065 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce03fff700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce0c0f9940 0x7fce0c1075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce0b7fe700 1 -- 172.21.15.78:0/295670065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce0c0f9940 msgr2=0x7fce0c1075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:08.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce0b7fe700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce0c0f9940 0x7fce0c1075b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce0b7fe700 1 -- 172.21.15.78:0/295670065 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc005040 con 0x7fce0c060870 2023-12-21T03:54:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce03fff700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce0c0f9940 0x7fce0c1075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:54:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.006+0000 7fce0b7fe700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce0c060870 0x7fce0c1070c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fcdfc009830 tx=0x7fcdfc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.007+0000 7fce08ff9700 1 -- 172.21.15.78:0/295670065 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdfc018070 con 0x7fce0c060870 2023-12-21T03:54:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.007+0000 7fce08ff9700 1 -- 172.21.15.78:0/295670065 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdfc013410 con 0x7fce0c060870 2023-12-21T03:54:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.007+0000 7fce11e06700 1 -- 172.21.15.78:0/295670065 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0c108720 con 0x7fce0c060870 2023-12-21T03:54:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.007+0000 7fce08ff9700 1 -- 172.21.15.78:0/295670065 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdfc01c620 con 0x7fce0c060870 2023-12-21T03:54:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.007+0000 7fce11e06700 1 -- 172.21.15.78:0/295670065 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0c108c80 con 0x7fce0c060870 2023-12-21T03:54:08.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.008+0000 7fce11e06700 1 -- 172.21.15.78:0/295670065 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c059100 con 0x7fce0c060870 2023-12-21T03:54:08.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.008+0000 7fce08ff9700 1 -- 172.21.15.78:0/295670065 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcdfc01c8e0 con 0x7fce0c060870 2023-12-21T03:54:08.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.009+0000 7fce08ff9700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdf805c2a0 0x7fcdf805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:08.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.009+0000 7fce08ff9700 1 -- 172.21.15.78:0/295670065 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fcdfc077dc0 con 0x7fce0c060870 2023-12-21T03:54:08.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.009+0000 7fce03fff700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdf805c2a0 0x7fcdf805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:08.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.009+0000 7fce03fff700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdf805c2a0 0x7fcdf805e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fce0c060640 tx=0x7fcdf4008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:08.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.012+0000 7fce08ff9700 1 -- 172.21.15.78:0/295670065 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcdfc04ae50 con 0x7fce0c060870 2023-12-21T03:54:08.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.219+0000 7fce11e06700 1 -- 172.21.15.78:0/295670065 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fce0c045ad0 con 0x7fce0c060870 2023-12-21T03:54:08.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.219+0000 7fce08ff9700 1 -- 172.21.15.78:0/295670065 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fcdfc04e470 con 0x7fce0c060870 2023-12-21T03:54:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:08.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.221+0000 7fce11e06700 1 -- 172.21.15.78:0/295670065 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdf805c2a0 msgr2=0x7fcdf805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:08.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.221+0000 7fce11e06700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdf805c2a0 0x7fcdf805e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fce0c060640 tx=0x7fcdf4008040).stop 2023-12-21T03:54:08.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.221+0000 7fce11e06700 1 -- 172.21.15.78:0/295670065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce0c060870 msgr2=0x7fce0c1070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:08.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.221+0000 7fce11e06700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce0c060870 0x7fce0c1070c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fcdfc009830 tx=0x7fcdfc009b30).stop 2023-12-21T03:54:08.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.221+0000 7fce11e06700 1 -- 172.21.15.78:0/295670065 shutdown_connections 2023-12-21T03:54:08.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.221+0000 7fce11e06700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fcdf805c2a0 0x7fcdf805e760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:08.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.221+0000 7fce11e06700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce0c060870 0x7fce0c1070c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:08.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.221+0000 7fce11e06700 1 --2- 172.21.15.78:0/295670065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce0c0f9940 0x7fce0c1075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:08.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.221+0000 7fce11e06700 1 -- 172.21.15.78:0/295670065 >> 172.21.15.78:0/295670065 conn(0x7fce0c0f0e20 msgr2=0x7fce0c0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:08.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.221+0000 7fce11e06700 1 -- 172.21.15.78:0/295670065 shutdown_connections 2023-12-21T03:54:08.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:08.221+0000 7fce11e06700 1 -- 172.21.15.78:0/295670065 wait complete. 2023-12-21T03:54:08.558 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:08.220350+0000 mon.smithi078 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.78:0/295670065' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:08.576 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:08.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:08.220350+0000 mon.smithi078 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.78:0/295670065' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:09.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:07.661191+0000 mgr.smithi078.zidbgq (mgr.14162) 496 : cluster [DBG] pgmap v464: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:09.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:07.661191+0000 mgr.smithi078.zidbgq (mgr.14162) 496 : cluster [DBG] pgmap v464: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:11.578 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:11.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:09.661616+0000 mgr.smithi078.zidbgq (mgr.14162) 497 : cluster [DBG] pgmap v465: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:11.814 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:11.815 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:09.661616+0000 mgr.smithi078.zidbgq (mgr.14162) 497 : cluster [DBG] pgmap v465: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.748+0000 7fca939c5700 1 -- 172.21.15.78:0/3849485821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca8c0f6960 msgr2=0x7fca8c0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.748+0000 7fca939c5700 1 --2- 172.21.15.78:0/3849485821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca8c0f6960 0x7fca8c0f6dc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fca88004660 tx=0x7fca88009b30).stop 2023-12-21T03:54:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.748+0000 7fca939c5700 1 -- 172.21.15.78:0/3849485821 shutdown_connections 2023-12-21T03:54:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.748+0000 7fca939c5700 1 --2- 172.21.15.78:0/3849485821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca8c0f6960 0x7fca8c0f6dc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.748+0000 7fca939c5700 1 --2- 172.21.15.78:0/3849485821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca8c0f56c0 0x7fca8c0f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.748+0000 7fca939c5700 1 -- 172.21.15.78:0/3849485821 >> 172.21.15.78:0/3849485821 conn(0x7fca8c0f0e20 msgr2=0x7fca8c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.748+0000 7fca939c5700 1 -- 172.21.15.78:0/3849485821 shutdown_connections 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.748+0000 7fca939c5700 1 -- 172.21.15.78:0/3849485821 wait complete. 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca939c5700 1 Processor -- start 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca939c5700 1 -- start start 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca939c5700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca8c0f56c0 0x7fca8c1070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca939c5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca8c0f6960 0x7fca8c1075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca939c5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c107b70 con 0x7fca8c0f6960 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca939c5700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c107cb0 con 0x7fca8c0f56c0 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca90f60700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca8c0f6960 0x7fca8c1075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca90f60700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca8c0f6960 0x7fca8c1075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52286/0 (socket says 172.21.15.78:52286) 2023-12-21T03:54:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca90f60700 1 -- 172.21.15.78:0/2136001351 learned_addr learned my addr 172.21.15.78:0/2136001351 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:12.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca91761700 1 --2- 172.21.15.78:0/2136001351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca8c0f56c0 0x7fca8c1070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:12.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca90f60700 1 -- 172.21.15.78:0/2136001351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca8c0f56c0 msgr2=0x7fca8c1070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:12.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca90f60700 1 --2- 172.21.15.78:0/2136001351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca8c0f56c0 0x7fca8c1070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:12.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.749+0000 7fca90f60700 1 -- 172.21.15.78:0/2136001351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca88005040 con 0x7fca8c0f6960 2023-12-21T03:54:12.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.750+0000 7fca90f60700 1 --2- 172.21.15.78:0/2136001351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca8c0f6960 0x7fca8c1075c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fca88009830 tx=0x7fca88009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:12.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.750+0000 7fca81ffb700 1 -- 172.21.15.78:0/2136001351 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca88018070 con 0x7fca8c0f6960 2023-12-21T03:54:12.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.750+0000 7fca939c5700 1 -- 172.21.15.78:0/2136001351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8c1085f0 con 0x7fca8c0f6960 2023-12-21T03:54:12.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.750+0000 7fca81ffb700 1 -- 172.21.15.78:0/2136001351 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca88013410 con 0x7fca8c0f6960 2023-12-21T03:54:12.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.750+0000 7fca81ffb700 1 -- 172.21.15.78:0/2136001351 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca8801c400 con 0x7fca8c0f6960 2023-12-21T03:54:12.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.750+0000 7fca939c5700 1 -- 172.21.15.78:0/2136001351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8c108b50 con 0x7fca8c0f6960 2023-12-21T03:54:12.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.751+0000 7fca939c5700 1 -- 172.21.15.78:0/2136001351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8c045ad0 con 0x7fca8c0f6960 2023-12-21T03:54:12.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.751+0000 7fca81ffb700 1 -- 172.21.15.78:0/2136001351 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fca8801d420 con 0x7fca8c0f6960 2023-12-21T03:54:12.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.752+0000 7fca81ffb700 1 --2- 172.21.15.78:0/2136001351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fca7805c2a0 0x7fca7805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:12.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.752+0000 7fca81ffb700 1 -- 172.21.15.78:0/2136001351 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fca88077a50 con 0x7fca8c0f6960 2023-12-21T03:54:12.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.752+0000 7fca91761700 1 --2- 172.21.15.78:0/2136001351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fca7805c2a0 0x7fca7805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:12.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.752+0000 7fca91761700 1 --2- 172.21.15.78:0/2136001351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fca7805c2a0 0x7fca7805e760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fca8c0f67c0 tx=0x7fca7c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.755+0000 7fca81ffb700 1 -- 172.21.15.78:0/2136001351 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca8804a9f0 con 0x7fca8c0f6960 2023-12-21T03:54:12.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.963+0000 7fca939c5700 1 -- 172.21.15.78:0/2136001351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fca8c109250 con 0x7fca8c0f6960 2023-12-21T03:54:12.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.964+0000 7fca81ffb700 1 -- 172.21.15.78:0/2136001351 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fca8804e010 con 0x7fca8c0f6960 2023-12-21T03:54:12.965 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:12.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.965+0000 7fca939c5700 1 -- 172.21.15.78:0/2136001351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fca7805c2a0 msgr2=0x7fca7805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.965+0000 7fca939c5700 1 --2- 172.21.15.78:0/2136001351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fca7805c2a0 0x7fca7805e760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fca8c0f67c0 tx=0x7fca7c009040).stop 2023-12-21T03:54:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.965+0000 7fca939c5700 1 -- 172.21.15.78:0/2136001351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca8c0f6960 msgr2=0x7fca8c1075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.965+0000 7fca939c5700 1 --2- 172.21.15.78:0/2136001351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca8c0f6960 0x7fca8c1075c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fca88009830 tx=0x7fca88009b30).stop 2023-12-21T03:54:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.965+0000 7fca939c5700 1 -- 172.21.15.78:0/2136001351 shutdown_connections 2023-12-21T03:54:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.965+0000 7fca939c5700 1 --2- 172.21.15.78:0/2136001351 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fca7805c2a0 0x7fca7805e760 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.965+0000 7fca939c5700 1 --2- 172.21.15.78:0/2136001351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca8c0f56c0 0x7fca8c1070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.965+0000 7fca939c5700 1 --2- 172.21.15.78:0/2136001351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca8c0f6960 0x7fca8c1075c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.965+0000 7fca939c5700 1 -- 172.21.15.78:0/2136001351 >> 172.21.15.78:0/2136001351 conn(0x7fca8c0f0e20 msgr2=0x7fca8c0603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.965+0000 7fca939c5700 1 -- 172.21.15.78:0/2136001351 shutdown_connections 2023-12-21T03:54:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:12.965+0000 7fca939c5700 1 -- 172.21.15.78:0/2136001351 wait complete. 2023-12-21T03:54:13.334 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:13.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:11.661961+0000 mgr.smithi078.zidbgq (mgr.14162) 498 : cluster [DBG] pgmap v466: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:13.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:12.965145+0000 mon.smithi078 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.78:0/2136001351' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:13.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:11.661961+0000 mgr.smithi078.zidbgq (mgr.14162) 498 : cluster [DBG] pgmap v466: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:13.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:12.965145+0000 mon.smithi078 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.78:0/2136001351' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:15.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:13.662261+0000 mgr.smithi078.zidbgq (mgr.14162) 499 : cluster [DBG] pgmap v467: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:13.662261+0000 mgr.smithi078.zidbgq (mgr.14162) 499 : cluster [DBG] pgmap v467: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:16.334 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:16.575 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.564+0000 7f08266b6700 1 -- 172.21.15.78:0/1211528556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08200f6960 msgr2=0x7f08200f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.564+0000 7f08266b6700 1 --2- 172.21.15.78:0/1211528556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08200f6960 0x7f08200f6dc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f0814004660 tx=0x7f0814009b30).stop 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.564+0000 7f08266b6700 1 -- 172.21.15.78:0/1211528556 shutdown_connections 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.564+0000 7f08266b6700 1 --2- 172.21.15.78:0/1211528556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08200f6960 0x7f08200f6dc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.564+0000 7f08266b6700 1 --2- 172.21.15.78:0/1211528556 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08200f56c0 0x7f08200f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.564+0000 7f08266b6700 1 -- 172.21.15.78:0/1211528556 >> 172.21.15.78:0/1211528556 conn(0x7f08200f0e20 msgr2=0x7f08200f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.564+0000 7f08266b6700 1 -- 172.21.15.78:0/1211528556 shutdown_connections 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.564+0000 7f08266b6700 1 -- 172.21.15.78:0/1211528556 wait complete. 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.565+0000 7f08266b6700 1 Processor -- start 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.565+0000 7f08266b6700 1 -- start start 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.565+0000 7f08266b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08200f56c0 0x7f08201070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.565+0000 7f08266b6700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08200f6960 0x7f08201075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.565+0000 7f08266b6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0820107b80 con 0x7f08200f56c0 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.565+0000 7f08266b6700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0820107cc0 con 0x7f08200f6960 2023-12-21T03:54:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.565+0000 7f081ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08200f56c0 0x7f08201070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.565+0000 7f081ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08200f56c0 0x7f08201070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52304/0 (socket says 172.21.15.78:52304) 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.565+0000 7f081ffff700 1 -- 172.21.15.78:0/3517679985 learned_addr learned my addr 172.21.15.78:0/3517679985 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.566+0000 7f081f7fe700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08200f6960 0x7f08201075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.566+0000 7f081ffff700 1 -- 172.21.15.78:0/3517679985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08200f6960 msgr2=0x7f08201075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.566+0000 7f081ffff700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08200f6960 0x7f08201075d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.566+0000 7f081ffff700 1 -- 172.21.15.78:0/3517679985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0814005040 con 0x7f08200f56c0 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.566+0000 7f081f7fe700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08200f6960 0x7f08201075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.566+0000 7f081ffff700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08200f56c0 0x7f08201070e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f081000c530 tx=0x7f0810007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.566+0000 7f081cff9700 1 -- 172.21.15.78:0/3517679985 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f081000f650 con 0x7f08200f56c0 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.566+0000 7f081cff9700 1 -- 172.21.15.78:0/3517679985 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f081000fc60 con 0x7f08200f56c0 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.566+0000 7f08266b6700 1 -- 172.21.15.78:0/3517679985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0820108660 con 0x7f08200f56c0 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.566+0000 7f081cff9700 1 -- 172.21.15.78:0/3517679985 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0810012d20 con 0x7f08200f56c0 2023-12-21T03:54:17.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.566+0000 7f08266b6700 1 -- 172.21.15.78:0/3517679985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0820108c20 con 0x7f08200f56c0 2023-12-21T03:54:17.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.567+0000 7f08266b6700 1 -- 172.21.15.78:0/3517679985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0820045ad0 con 0x7f08200f56c0 2023-12-21T03:54:17.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.567+0000 7f081cff9700 1 -- 172.21.15.78:0/3517679985 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0810019030 con 0x7f08200f56c0 2023-12-21T03:54:17.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.568+0000 7f081cff9700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f080805c2a0 0x7f080805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:17.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.568+0000 7f081cff9700 1 -- 172.21.15.78:0/3517679985 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0810075830 con 0x7f08200f56c0 2023-12-21T03:54:17.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.568+0000 7f081f7fe700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f080805c2a0 0x7f080805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:17.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.568+0000 7f081f7fe700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f080805c2a0 0x7f080805e760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f0814013580 tx=0x7f0814009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:17.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.571+0000 7f081cff9700 1 -- 172.21.15.78:0/3517679985 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0810048880 con 0x7f08200f56c0 2023-12-21T03:54:17.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:15.662723+0000 mgr.smithi078.zidbgq (mgr.14162) 500 : cluster [DBG] pgmap v468: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:17.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.779+0000 7f08266b6700 1 -- 172.21.15.78:0/3517679985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f08201092c0 con 0x7f08200f56c0 2023-12-21T03:54:17.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.779+0000 7f081cff9700 1 -- 172.21.15.78:0/3517679985 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f0810017070 con 0x7f08200f56c0 2023-12-21T03:54:17.781 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:17.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.781+0000 7f08266b6700 1 -- 172.21.15.78:0/3517679985 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f080805c2a0 msgr2=0x7f080805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:17.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.781+0000 7f08266b6700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f080805c2a0 0x7f080805e760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f0814013580 tx=0x7f0814009b30).stop 2023-12-21T03:54:17.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.781+0000 7f08266b6700 1 -- 172.21.15.78:0/3517679985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08200f56c0 msgr2=0x7f08201070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:17.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.781+0000 7f08266b6700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08200f56c0 0x7f08201070e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f081000c530 tx=0x7f0810007e10).stop 2023-12-21T03:54:17.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.781+0000 7f08266b6700 1 -- 172.21.15.78:0/3517679985 shutdown_connections 2023-12-21T03:54:17.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.781+0000 7f08266b6700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f080805c2a0 0x7f080805e760 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:17.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.781+0000 7f08266b6700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08200f56c0 0x7f08201070e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:17.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.781+0000 7f08266b6700 1 --2- 172.21.15.78:0/3517679985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08200f6960 0x7f08201075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:17.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.781+0000 7f08266b6700 1 -- 172.21.15.78:0/3517679985 >> 172.21.15.78:0/3517679985 conn(0x7f08200f0e20 msgr2=0x7f08200603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:17.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.781+0000 7f08266b6700 1 -- 172.21.15.78:0/3517679985 shutdown_connections 2023-12-21T03:54:17.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:17.781+0000 7f08266b6700 1 -- 172.21.15.78:0/3517679985 wait complete. 2023-12-21T03:54:17.810 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:15.662723+0000 mgr.smithi078.zidbgq (mgr.14162) 500 : cluster [DBG] pgmap v468: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:18.166 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:18.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:18 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:17.780670+0000 mon.smithi078 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.78:0/3517679985' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:18.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:18 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:17.780670+0000 mon.smithi078 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.78:0/3517679985' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:19.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:17.663051+0000 mgr.smithi078.zidbgq (mgr.14162) 501 : cluster [DBG] pgmap v469: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:19.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:17.663051+0000 mgr.smithi078.zidbgq (mgr.14162) 501 : cluster [DBG] pgmap v469: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:21.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:21.399 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:21.509 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:19.663411+0000 mgr.smithi078.zidbgq (mgr.14162) 502 : cluster [DBG] pgmap v470: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:21.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:19.663411+0000 mgr.smithi078.zidbgq (mgr.14162) 502 : cluster [DBG] pgmap v470: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:22.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.350+0000 7f1d6dc44700 1 -- 172.21.15.78:0/3426368489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680f67c0 msgr2=0x7f1d680f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.350+0000 7f1d6dc44700 1 --2- 172.21.15.78:0/3426368489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680f67c0 0x7f1d680f6be0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f1d58004660 tx=0x7f1d58009b30).stop 2023-12-21T03:54:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.350+0000 7f1d6dc44700 1 -- 172.21.15.78:0/3426368489 shutdown_connections 2023-12-21T03:54:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.350+0000 7f1d6dc44700 1 --2- 172.21.15.78:0/3426368489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d680f7a60 0x7f1d680f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:54:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.350+0000 7f1d6dc44700 1 --2- 172.21.15.78:0/3426368489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680f67c0 0x7f1d680f6be0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.350+0000 7f1d6dc44700 1 -- 172.21.15.78:0/3426368489 >> 172.21.15.78:0/3426368489 conn(0x7f1d680f1f20 msgr2=0x7f1d680f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.351+0000 7f1d6dc44700 1 -- 172.21.15.78:0/3426368489 shutdown_connections 2023-12-21T03:54:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.351+0000 7f1d6dc44700 1 -- 172.21.15.78:0/3426368489 wait complete. 2023-12-21T03:54:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.351+0000 7f1d6dc44700 1 Processor -- start 2023-12-21T03:54:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.351+0000 7f1d6dc44700 1 -- start start 2023-12-21T03:54:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.351+0000 7f1d6dc44700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680f67c0 0x7f1d681070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.351+0000 7f1d6dc44700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d680f7a60 0x7f1d681075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.351+0000 7f1d6dc44700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d68107c00 con 0x7f1d680f67c0 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.351+0000 7f1d6dc44700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d681084b0 con 0x7f1d680f7a60 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d677fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680f67c0 0x7f1d681070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d677fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680f67c0 0x7f1d681070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44136/0 (socket says 172.21.15.78:44136) 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d677fe700 1 -- 172.21.15.78:0/144596245 learned_addr learned my addr 172.21.15.78:0/144596245 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d677fe700 1 -- 172.21.15.78:0/144596245 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d680f7a60 msgr2=0x7f1d681075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d66ffd700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d680f7a60 0x7f1d681075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d677fe700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d680f7a60 0x7f1d681075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d677fe700 1 -- 172.21.15.78:0/144596245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d58005040 con 0x7f1d680f67c0 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d66ffd700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d680f7a60 0x7f1d681075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d677fe700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680f67c0 0x7f1d681070d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f1d58009f50 tx=0x7f1d58009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d4ffff700 1 -- 172.21.15.78:0/144596245 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d58018070 con 0x7f1d680f67c0 2023-12-21T03:54:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d6dc44700 1 -- 172.21.15.78:0/144596245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d68108730 con 0x7f1d680f67c0 2023-12-21T03:54:22.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d4ffff700 1 -- 172.21.15.78:0/144596245 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d58013410 con 0x7f1d680f67c0 2023-12-21T03:54:22.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d4ffff700 1 -- 172.21.15.78:0/144596245 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d5801c620 con 0x7f1d680f67c0 2023-12-21T03:54:22.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.352+0000 7f1d6dc44700 1 -- 172.21.15.78:0/144596245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d68108c90 con 0x7f1d680f67c0 2023-12-21T03:54:22.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.353+0000 7f1d6dc44700 1 -- 172.21.15.78:0/144596245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d68059100 con 0x7f1d680f67c0 2023-12-21T03:54:22.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.354+0000 7f1d4ffff700 1 -- 172.21.15.78:0/144596245 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1d5801d420 con 0x7f1d680f67c0 2023-12-21T03:54:22.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.354+0000 7f1d4ffff700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1d5005c2a0 0x7f1d5005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:22.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.354+0000 7f1d4ffff700 1 -- 172.21.15.78:0/144596245 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1d58077a40 con 0x7f1d680f67c0 2023-12-21T03:54:22.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.354+0000 7f1d66ffd700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1d5005c2a0 0x7f1d5005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:22.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.354+0000 7f1d66ffd700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1d5005c2a0 0x7f1d5005e760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f1d5c00a7e0 tx=0x7f1d5c008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:22.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.358+0000 7f1d4ffff700 1 -- 172.21.15.78:0/144596245 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d5804aa40 con 0x7f1d680f67c0 2023-12-21T03:54:22.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.565+0000 7f1d6dc44700 1 -- 172.21.15.78:0/144596245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1d68045ad0 con 0x7f1d680f67c0 2023-12-21T03:54:22.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.565+0000 7f1d4ffff700 1 -- 172.21.15.78:0/144596245 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f1d5804e060 con 0x7f1d680f67c0 2023-12-21T03:54:22.566 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.567+0000 7f1d6dc44700 1 -- 172.21.15.78:0/144596245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1d5005c2a0 msgr2=0x7f1d5005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.567+0000 7f1d6dc44700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1d5005c2a0 0x7f1d5005e760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f1d5c00a7e0 tx=0x7f1d5c008040).stop 2023-12-21T03:54:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.567+0000 7f1d6dc44700 1 -- 172.21.15.78:0/144596245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680f67c0 msgr2=0x7f1d681070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.567+0000 7f1d6dc44700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680f67c0 0x7f1d681070d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f1d58009f50 tx=0x7f1d58009b30).stop 2023-12-21T03:54:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.567+0000 7f1d6dc44700 1 -- 172.21.15.78:0/144596245 shutdown_connections 2023-12-21T03:54:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.567+0000 7f1d6dc44700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f1d5005c2a0 0x7f1d5005e760 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.567+0000 7f1d6dc44700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680f67c0 0x7f1d681070d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.567+0000 7f1d6dc44700 1 --2- 172.21.15.78:0/144596245 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d680f7a60 0x7f1d681075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.567+0000 7f1d6dc44700 1 -- 172.21.15.78:0/144596245 >> 172.21.15.78:0/144596245 conn(0x7f1d680f1f20 msgr2=0x7f1d680fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.567+0000 7f1d6dc44700 1 -- 172.21.15.78:0/144596245 shutdown_connections 2023-12-21T03:54:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:22.567+0000 7f1d6dc44700 1 -- 172.21.15.78:0/144596245 wait complete. 2023-12-21T03:54:22.974 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:23.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:21.663762+0000 mgr.smithi078.zidbgq (mgr.14162) 503 : cluster [DBG] pgmap v471: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:23.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:22.566466+0000 mon.smithi078 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.78:0/144596245' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:21.663762+0000 mgr.smithi078.zidbgq (mgr.14162) 503 : cluster [DBG] pgmap v471: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:22.566466+0000 mon.smithi078 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.78:0/144596245' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:25.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:23.664208+0000 mgr.smithi078.zidbgq (mgr.14162) 504 : cluster [DBG] pgmap v472: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:23.664208+0000 mgr.smithi078.zidbgq (mgr.14162) 504 : cluster [DBG] pgmap v472: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:25.975 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:26.212 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.187+0000 7f203c007700 1 -- 172.21.15.78:0/1006063068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20340f56e0 msgr2=0x7f20340f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.187+0000 7f203c007700 1 --2- 172.21.15.78:0/1006063068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20340f56e0 0x7f20340f5b00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f2024004660 tx=0x7f2024009b30).stop 2023-12-21T03:54:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.187+0000 7f203c007700 1 -- 172.21.15.78:0/1006063068 shutdown_connections 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.187+0000 7f203c007700 1 --2- 172.21.15.78:0/1006063068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20340f6980 0x7f20340f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.187+0000 7f203c007700 1 --2- 172.21.15.78:0/1006063068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20340f56e0 0x7f20340f5b00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.187+0000 7f203c007700 1 -- 172.21.15.78:0/1006063068 >> 172.21.15.78:0/1006063068 conn(0x7f20340f0e20 msgr2=0x7f20340f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.187+0000 7f203c007700 1 -- 172.21.15.78:0/1006063068 shutdown_connections 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.187+0000 7f203c007700 1 -- 172.21.15.78:0/1006063068 wait complete. 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f203c007700 1 Processor -- start 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f203c007700 1 -- start start 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f203c007700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20340f56e0 0x7f203410c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f203c007700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20340f6980 0x7f203410e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f203c007700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2034060810 con 0x7f20340f56e0 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f203c007700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2034060950 con 0x7f20340f6980 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f2039da3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20340f56e0 0x7f203410c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f2039da3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20340f56e0 0x7f203410c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44152/0 (socket says 172.21.15.78:44152) 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f2039da3700 1 -- 172.21.15.78:0/4056721815 learned_addr learned my addr 172.21.15.78:0/4056721815 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f20395a2700 1 --2- 172.21.15.78:0/4056721815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20340f6980 0x7f203410e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f2039da3700 1 -- 172.21.15.78:0/4056721815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20340f6980 msgr2=0x7f203410e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f2039da3700 1 --2- 172.21.15.78:0/4056721815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20340f6980 0x7f203410e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.188+0000 7f2039da3700 1 -- 172.21.15.78:0/4056721815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2024005040 con 0x7f20340f56e0 2023-12-21T03:54:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.189+0000 7f2039da3700 1 --2- 172.21.15.78:0/4056721815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20340f56e0 0x7f203410c220 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f2024009f50 tx=0x7f2024009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.189+0000 7f202a7fc700 1 -- 172.21.15.78:0/4056721815 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2024018070 con 0x7f20340f56e0 2023-12-21T03:54:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.189+0000 7f202a7fc700 1 -- 172.21.15.78:0/4056721815 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2024013410 con 0x7f20340f56e0 2023-12-21T03:54:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.189+0000 7f203c007700 1 -- 172.21.15.78:0/4056721815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203410ec10 con 0x7f20340f56e0 2023-12-21T03:54:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.189+0000 7f202a7fc700 1 -- 172.21.15.78:0/4056721815 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f202401c620 con 0x7f20340f56e0 2023-12-21T03:54:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.189+0000 7f203c007700 1 -- 172.21.15.78:0/4056721815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203410efe0 con 0x7f20340f56e0 2023-12-21T03:54:27.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.190+0000 7f203c007700 1 -- 172.21.15.78:0/4056721815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2034045ad0 con 0x7f20340f56e0 2023-12-21T03:54:27.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.190+0000 7f202a7fc700 1 -- 172.21.15.78:0/4056721815 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f202401d420 con 0x7f20340f56e0 2023-12-21T03:54:27.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.190+0000 7f202a7fc700 1 --2- 172.21.15.78:0/4056721815 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f202005c2a0 0x7f202005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:27.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.190+0000 7f202a7fc700 1 -- 172.21.15.78:0/4056721815 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2024077a00 con 0x7f20340f56e0 2023-12-21T03:54:27.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.190+0000 7f20395a2700 1 --2- 172.21.15.78:0/4056721815 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f202005c2a0 0x7f202005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:27.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.191+0000 7f20395a2700 1 --2- 172.21.15.78:0/4056721815 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f202005c2a0 0x7f202005e760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f2030007fd0 tx=0x7f2030009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.194+0000 7f202a7fc700 1 -- 172.21.15.78:0/4056721815 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f202404aa00 con 0x7f20340f56e0 2023-12-21T03:54:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.402+0000 7f203c007700 1 -- 172.21.15.78:0/4056721815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f20340fdf40 con 0x7f20340f56e0 2023-12-21T03:54:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.403+0000 7f202a7fc700 1 -- 172.21.15.78:0/4056721815 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f202404e020 con 0x7f20340f56e0 2023-12-21T03:54:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:27.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.404+0000 7f203c007700 1 -- 172.21.15.78:0/4056721815 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f202005c2a0 msgr2=0x7f202005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:27.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.404+0000 7f203c007700 1 --2- 172.21.15.78:0/4056721815 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f202005c2a0 0x7f202005e760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f2030007fd0 tx=0x7f2030009040).stop 2023-12-21T03:54:27.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.404+0000 7f203c007700 1 -- 172.21.15.78:0/4056721815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20340f56e0 msgr2=0x7f203410c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:27.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.404+0000 7f203c007700 1 --2- 172.21.15.78:0/4056721815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20340f56e0 0x7f203410c220 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f2024009f50 tx=0x7f2024009b30).stop 2023-12-21T03:54:27.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.404+0000 7f203c007700 1 -- 172.21.15.78:0/4056721815 shutdown_connections 2023-12-21T03:54:27.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.404+0000 7f203c007700 1 --2- 172.21.15.78:0/4056721815 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f202005c2a0 0x7f202005e760 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:27.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.404+0000 7f203c007700 1 --2- 172.21.15.78:0/4056721815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20340f56e0 0x7f203410c220 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:27.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.404+0000 7f203c007700 1 --2- 172.21.15.78:0/4056721815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20340f6980 0x7f203410e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:27.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.404+0000 7f203c007700 1 -- 172.21.15.78:0/4056721815 >> 172.21.15.78:0/4056721815 conn(0x7f20340f0e20 msgr2=0x7f20340fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:27.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.404+0000 7f203c007700 1 -- 172.21.15.78:0/4056721815 shutdown_connections 2023-12-21T03:54:27.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:27.404+0000 7f203c007700 1 -- 172.21.15.78:0/4056721815 wait complete. 2023-12-21T03:54:27.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:25.664629+0000 mgr.smithi078.zidbgq (mgr.14162) 505 : cluster [DBG] pgmap v473: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:27.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:27.404064+0000 mon.smithi078 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.78:0/4056721815' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:27.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:25.664629+0000 mgr.smithi078.zidbgq (mgr.14162) 505 : cluster [DBG] pgmap v473: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:27.773 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:27.404064+0000 mon.smithi078 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.78:0/4056721815' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:27.793 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:29.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:27.664972+0000 mgr.smithi078.zidbgq (mgr.14162) 506 : cluster [DBG] pgmap v474: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:27.664972+0000 mgr.smithi078.zidbgq (mgr.14162) 506 : cluster [DBG] pgmap v474: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:30.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:29.665301+0000 mgr.smithi078.zidbgq (mgr.14162) 507 : cluster [DBG] pgmap v475: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:30.567840+0000 mon.smithi078 (mon.0) 536 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:54:31.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:30.568283+0000 mon.smithi078 (mon.0) 537 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:54:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:29.665301+0000 mgr.smithi078.zidbgq (mgr.14162) 507 : cluster [DBG] pgmap v475: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:30.567840+0000 mon.smithi078 (mon.0) 536 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:54:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:30.568283+0000 mon.smithi078 (mon.0) 537 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:54:32.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.044+0000 7f02dffff700 1 -- 172.21.15.78:0/133516483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02e00f9770 msgr2=0x7f02e00fbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:32.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.044+0000 7f02dffff700 1 --2- 172.21.15.78:0/133516483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02e00f9770 0x7f02e00fbbc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f02d0004660 tx=0x7f02d0009b30).stop 2023-12-21T03:54:32.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.044+0000 7f02dffff700 1 -- 172.21.15.78:0/133516483 shutdown_connections 2023-12-21T03:54:32.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.044+0000 7f02dffff700 1 --2- 172.21.15.78:0/133516483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02e00f9770 0x7f02e00fbbc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:32.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.044+0000 7f02dffff700 1 --2- 172.21.15.78:0/133516483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0060870 0x7f02e00f91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:54:32.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.044+0000 7f02dffff700 1 -- 172.21.15.78:0/133516483 >> 172.21.15.78:0/133516483 conn(0x7f02e00f0c70 msgr2=0x7f02e00f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:32.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.044+0000 7f02dffff700 1 -- 172.21.15.78:0/133516483 shutdown_connections 2023-12-21T03:54:32.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.044+0000 7f02dffff700 1 -- 172.21.15.78:0/133516483 wait complete. 2023-12-21T03:54:32.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.045+0000 7f02dffff700 1 Processor -- start 2023-12-21T03:54:32.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.045+0000 7f02dffff700 1 -- start start 2023-12-21T03:54:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.045+0000 7f02dffff700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0060870 0x7f02e0106f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.045+0000 7f02dffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02e00f9770 0x7f02e0107400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.045+0000 7f02dffff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e0107a40 con 0x7f02e00f9770 2023-12-21T03:54:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.045+0000 7f02dffff700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e01082f0 con 0x7f02e0060870 2023-12-21T03:54:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02de7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02e00f9770 0x7f02e0107400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02de7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02e00f9770 0x7f02e0107400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53764/0 (socket says 172.21.15.78:53764) 2023-12-21T03:54:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02de7fc700 1 -- 172.21.15.78:0/2958236189 learned_addr learned my addr 172.21.15.78:0/2958236189 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:32.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02deffd700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0060870 0x7f02e0106f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:32.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02de7fc700 1 -- 172.21.15.78:0/2958236189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0060870 msgr2=0x7f02e0106f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:32.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02de7fc700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0060870 0x7f02e0106f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02de7fc700 1 -- 172.21.15.78:0/2958236189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d0005040 con 0x7f02e00f9770 2023-12-21T03:54:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02deffd700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0060870 0x7f02e0106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:54:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02de7fc700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02e00f9770 0x7f02e0107400 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f02d0009f50 tx=0x7f02d0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02c77fe700 1 -- 172.21.15.78:0/2958236189 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02d0018070 con 0x7f02e00f9770 2023-12-21T03:54:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02c77fe700 1 -- 172.21.15.78:0/2958236189 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02d0013410 con 0x7f02e00f9770 2023-12-21T03:54:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02dffff700 1 -- 172.21.15.78:0/2958236189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e0108570 con 0x7f02e00f9770 2023-12-21T03:54:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02c77fe700 1 -- 172.21.15.78:0/2958236189 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02d001c620 con 0x7f02e00f9770 2023-12-21T03:54:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.046+0000 7f02dffff700 1 -- 172.21.15.78:0/2958236189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e0108ad0 con 0x7f02e00f9770 2023-12-21T03:54:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.047+0000 7f02dffff700 1 -- 172.21.15.78:0/2958236189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e0059100 con 0x7f02e00f9770 2023-12-21T03:54:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.047+0000 7f02c77fe700 1 -- 172.21.15.78:0/2958236189 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f02d001d420 con 0x7f02e00f9770 2023-12-21T03:54:32.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.048+0000 7f02c77fe700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f02c805c2b0 0x7f02c805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:32.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.048+0000 7f02deffd700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f02c805c2b0 0x7f02c805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:32.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.048+0000 7f02c77fe700 1 -- 172.21.15.78:0/2958236189 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f02d00779f0 con 0x7f02e00f9770 2023-12-21T03:54:32.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.048+0000 7f02deffd700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f02c805c2b0 0x7f02c805e770 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f02d400a7e0 tx=0x7f02d4008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:32.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.053+0000 7f02c77fe700 1 -- 172.21.15.78:0/2958236189 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02d0079e60 con 0x7f02e00f9770 2023-12-21T03:54:32.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.258+0000 7f02dffff700 1 -- 172.21.15.78:0/2958236189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f02e0045ad0 con 0x7f02e00f9770 2023-12-21T03:54:32.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.258+0000 7f02c77fe700 1 -- 172.21.15.78:0/2958236189 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f02e0045ad0 con 0x7f02e00f9770 2023-12-21T03:54:32.260 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.260+0000 7f02dffff700 1 -- 172.21.15.78:0/2958236189 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f02c805c2b0 msgr2=0x7f02c805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.260+0000 7f02dffff700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f02c805c2b0 0x7f02c805e770 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f02d400a7e0 tx=0x7f02d4008040).stop 2023-12-21T03:54:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.260+0000 7f02dffff700 1 -- 172.21.15.78:0/2958236189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02e00f9770 msgr2=0x7f02e0107400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.260+0000 7f02dffff700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02e00f9770 0x7f02e0107400 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f02d0009f50 tx=0x7f02d0009b30).stop 2023-12-21T03:54:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.260+0000 7f02dffff700 1 -- 172.21.15.78:0/2958236189 shutdown_connections 2023-12-21T03:54:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.260+0000 7f02dffff700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f02c805c2b0 0x7f02c805e770 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.260+0000 7f02dffff700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0060870 0x7f02e0106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.260+0000 7f02dffff700 1 --2- 172.21.15.78:0/2958236189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02e00f9770 0x7f02e0107400 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.260+0000 7f02dffff700 1 -- 172.21.15.78:0/2958236189 >> 172.21.15.78:0/2958236189 conn(0x7f02e00f0c70 msgr2=0x7f02e00f3060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.260+0000 7f02dffff700 1 -- 172.21.15.78:0/2958236189 shutdown_connections 2023-12-21T03:54:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:32.260+0000 7f02dffff700 1 -- 172.21.15.78:0/2958236189 wait complete. 2023-12-21T03:54:32.563 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:32.259920+0000 mon.smithi078 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.78:0/2958236189' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:32.639 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:32.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:32 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:32.259920+0000 mon.smithi078 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.78:0/2958236189' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:33.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:31.665614+0000 mgr.smithi078.zidbgq (mgr.14162) 508 : cluster [DBG] pgmap v476: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:33.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:31.665614+0000 mgr.smithi078.zidbgq (mgr.14162) 508 : cluster [DBG] pgmap v476: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:35.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:33.666084+0000 mgr.smithi078.zidbgq (mgr.14162) 509 : cluster [DBG] pgmap v477: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:35.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:33.666084+0000 mgr.smithi078.zidbgq (mgr.14162) 509 : cluster [DBG] pgmap v477: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:36.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.789+0000 7f8c892fe700 1 -- 172.21.15.78:0/2888544008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c840f6650 msgr2=0x7f8c840f6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:36.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.789+0000 7f8c892fe700 1 --2- 172.21.15.78:0/2888544008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c840f6650 0x7f8c840f6a70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f8c74004660 tx=0x7f8c74009b30).stop 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.789+0000 7f8c892fe700 1 -- 172.21.15.78:0/2888544008 shutdown_connections 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.789+0000 7f8c892fe700 1 --2- 172.21.15.78:0/2888544008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c840f78f0 0x7f8c840f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.789+0000 7f8c892fe700 1 --2- 172.21.15.78:0/2888544008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c840f6650 0x7f8c840f6a70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.789+0000 7f8c892fe700 1 -- 172.21.15.78:0/2888544008 >> 172.21.15.78:0/2888544008 conn(0x7f8c840f1db0 msgr2=0x7f8c840f41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.789+0000 7f8c892fe700 1 -- 172.21.15.78:0/2888544008 shutdown_connections 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.789+0000 7f8c892fe700 1 -- 172.21.15.78:0/2888544008 wait complete. 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.790+0000 7f8c892fe700 1 Processor -- start 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.790+0000 7f8c892fe700 1 -- start start 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.790+0000 7f8c892fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c840f6650 0x7f8c8406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.790+0000 7f8c892fe700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c840f78f0 0x7f8c84069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.790+0000 7f8c892fe700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8406a380 con 0x7f8c840f6650 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.790+0000 7f8c892fe700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8406a4c0 con 0x7f8c840f78f0 2023-12-21T03:54:36.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.790+0000 7f8c82ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c840f6650 0x7f8c8406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.790+0000 7f8c82ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c840f6650 0x7f8c8406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53780/0 (socket says 172.21.15.78:53780) 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.790+0000 7f8c82ffd700 1 -- 172.21.15.78:0/1181406746 learned_addr learned my addr 172.21.15.78:0/1181406746 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.791+0000 7f8c827fc700 1 --2- 172.21.15.78:0/1181406746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c840f78f0 0x7f8c84069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.791+0000 7f8c82ffd700 1 -- 172.21.15.78:0/1181406746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c840f78f0 msgr2=0x7f8c84069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.791+0000 7f8c82ffd700 1 --2- 172.21.15.78:0/1181406746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c840f78f0 0x7f8c84069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.791+0000 7f8c82ffd700 1 -- 172.21.15.78:0/1181406746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c74005040 con 0x7f8c840f6650 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.791+0000 7f8c82ffd700 1 --2- 172.21.15.78:0/1181406746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c840f6650 0x7f8c8406b9a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f8c74004e30 tx=0x7f8c74009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.791+0000 7f8c6b7fe700 1 -- 172.21.15.78:0/1181406746 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c74018070 con 0x7f8c840f6650 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.791+0000 7f8c6b7fe700 1 -- 172.21.15.78:0/1181406746 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c74013410 con 0x7f8c840f6650 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.791+0000 7f8c6b7fe700 1 -- 172.21.15.78:0/1181406746 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c7401de20 con 0x7f8c840f6650 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.791+0000 7f8c892fe700 1 -- 172.21.15.78:0/1181406746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8406a740 con 0x7f8c840f6650 2023-12-21T03:54:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.791+0000 7f8c892fe700 1 -- 172.21.15.78:0/1181406746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8406aca0 con 0x7f8c840f6650 2023-12-21T03:54:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.792+0000 7f8c892fe700 1 -- 172.21.15.78:0/1181406746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c84045ad0 con 0x7f8c840f6650 2023-12-21T03:54:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.792+0000 7f8c6b7fe700 1 -- 172.21.15.78:0/1181406746 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8c7401d420 con 0x7f8c840f6650 2023-12-21T03:54:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.793+0000 7f8c6b7fe700 1 --2- 172.21.15.78:0/1181406746 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8c6c05c2b0 0x7f8c6c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:36.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.793+0000 7f8c6b7fe700 1 -- 172.21.15.78:0/1181406746 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8c740776e0 con 0x7f8c840f6650 2023-12-21T03:54:36.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.793+0000 7f8c827fc700 1 --2- 172.21.15.78:0/1181406746 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8c6c05c2b0 0x7f8c6c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:36.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.793+0000 7f8c827fc700 1 --2- 172.21.15.78:0/1181406746 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8c6c05c2b0 0x7f8c6c05e770 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f8c78004e20 tx=0x7f8c7800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:36.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:36.797+0000 7f8c6b7fe700 1 -- 172.21.15.78:0/1181406746 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c7404a760 con 0x7f8c840f6650 2023-12-21T03:54:37.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.004+0000 7f8c892fe700 1 -- 172.21.15.78:0/1181406746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8c84027a00 con 0x7f8c840f6650 2023-12-21T03:54:37.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.005+0000 7f8c6b7fe700 1 -- 172.21.15.78:0/1181406746 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f8c7404dd80 con 0x7f8c840f6650 2023-12-21T03:54:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:37.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.006+0000 7f8c892fe700 1 -- 172.21.15.78:0/1181406746 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8c6c05c2b0 msgr2=0x7f8c6c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:37.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.007+0000 7f8c892fe700 1 --2- 172.21.15.78:0/1181406746 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8c6c05c2b0 0x7f8c6c05e770 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f8c78004e20 tx=0x7f8c7800b040).stop 2023-12-21T03:54:37.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.007+0000 7f8c892fe700 1 -- 172.21.15.78:0/1181406746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c840f6650 msgr2=0x7f8c8406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:37.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.007+0000 7f8c892fe700 1 --2- 172.21.15.78:0/1181406746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c840f6650 0x7f8c8406b9a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f8c74004e30 tx=0x7f8c74009b30).stop 2023-12-21T03:54:37.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.007+0000 7f8c892fe700 1 -- 172.21.15.78:0/1181406746 shutdown_connections 2023-12-21T03:54:37.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.007+0000 7f8c892fe700 1 --2- 172.21.15.78:0/1181406746 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8c6c05c2b0 0x7f8c6c05e770 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:37.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.007+0000 7f8c892fe700 1 --2- 172.21.15.78:0/1181406746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c840f6650 0x7f8c8406b9a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:37.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.007+0000 7f8c892fe700 1 --2- 172.21.15.78:0/1181406746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c840f78f0 0x7f8c84069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:37.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.007+0000 7f8c892fe700 1 -- 172.21.15.78:0/1181406746 >> 172.21.15.78:0/1181406746 conn(0x7f8c840f1db0 msgr2=0x7f8c840fb340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:37.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.007+0000 7f8c892fe700 1 -- 172.21.15.78:0/1181406746 shutdown_connections 2023-12-21T03:54:37.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:37.007+0000 7f8c892fe700 1 -- 172.21.15.78:0/1181406746 wait complete. 2023-12-21T03:54:37.439 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:37.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:35.666423+0000 mgr.smithi078.zidbgq (mgr.14162) 510 : cluster [DBG] pgmap v478: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:37.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:37.005196+0000 mon.smithi078 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.78:0/1181406746' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:35.666423+0000 mgr.smithi078.zidbgq (mgr.14162) 510 : cluster [DBG] pgmap v478: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:37.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:37.005196+0000 mon.smithi078 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.78:0/1181406746' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:39.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:37.666742+0000 mgr.smithi078.zidbgq (mgr.14162) 511 : cluster [DBG] pgmap v479: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:39.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:37.666742+0000 mgr.smithi078.zidbgq (mgr.14162) 511 : cluster [DBG] pgmap v479: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:40.440 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.618+0000 7f17c137b700 1 -- 172.21.15.78:0/2730340298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17bc0f5680 msgr2=0x7f17bc0f5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.618+0000 7f17c137b700 1 --2- 172.21.15.78:0/2730340298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17bc0f5680 0x7f17bc0f5aa0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f17b0004660 tx=0x7f17b0009b30).stop 2023-12-21T03:54:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.618+0000 7f17c137b700 1 -- 172.21.15.78:0/2730340298 shutdown_connections 2023-12-21T03:54:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.618+0000 7f17c137b700 1 --2- 172.21.15.78:0/2730340298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17bc0f5f90 0x7f17bc0f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.618+0000 7f17c137b700 1 --2- 172.21.15.78:0/2730340298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17bc0f5680 0x7f17bc0f5aa0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.618+0000 7f17c137b700 1 -- 172.21.15.78:0/2730340298 >> 172.21.15.78:0/2730340298 conn(0x7f17bc0f0e80 msgr2=0x7f17bc0f32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.618+0000 7f17c137b700 1 -- 172.21.15.78:0/2730340298 shutdown_connections 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.618+0000 7f17c137b700 1 -- 172.21.15.78:0/2730340298 wait complete. 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17c137b700 1 Processor -- start 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17c137b700 1 -- start start 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17c137b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17bc0f5680 0x7f17bc109cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17c137b700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17bc0f5f90 0x7f17bc10c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17c137b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc0609d0 con 0x7f17bc0f5680 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17c137b700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc060b10 con 0x7f17bc0f5f90 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17baffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17bc0f5680 0x7f17bc109cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17baffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17bc0f5680 0x7f17bc109cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47584/0 (socket says 172.21.15.78:47584) 2023-12-21T03:54:41.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17baffd700 1 -- 172.21.15.78:0/2230679542 learned_addr learned my addr 172.21.15.78:0/2230679542 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:41.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17ba7fc700 1 --2- 172.21.15.78:0/2230679542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17bc0f5f90 0x7f17bc10c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:41.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17baffd700 1 -- 172.21.15.78:0/2230679542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17bc0f5f90 msgr2=0x7f17bc10c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:41.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17baffd700 1 --2- 172.21.15.78:0/2230679542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17bc0f5f90 0x7f17bc10c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:41.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17baffd700 1 -- 172.21.15.78:0/2230679542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b0005040 con 0x7f17bc0f5680 2023-12-21T03:54:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.619+0000 7f17baffd700 1 --2- 172.21.15.78:0/2230679542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17bc0f5680 0x7f17bc109cd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f17b0009830 tx=0x7f17b0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.620+0000 7f17a37fe700 1 -- 172.21.15.78:0/2230679542 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17b0018070 con 0x7f17bc0f5680 2023-12-21T03:54:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.620+0000 7f17c137b700 1 -- 172.21.15.78:0/2230679542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17bc10c7d0 con 0x7f17bc0f5680 2023-12-21T03:54:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.620+0000 7f17a37fe700 1 -- 172.21.15.78:0/2230679542 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17b0013410 con 0x7f17bc0f5680 2023-12-21T03:54:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.620+0000 7f17a37fe700 1 -- 172.21.15.78:0/2230679542 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17b001c620 con 0x7f17bc0f5680 2023-12-21T03:54:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.620+0000 7f17c137b700 1 -- 172.21.15.78:0/2230679542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17bc10cd00 con 0x7f17bc0f5680 2023-12-21T03:54:41.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.621+0000 7f17c137b700 1 -- 172.21.15.78:0/2230679542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17bc045ad0 con 0x7f17bc0f5680 2023-12-21T03:54:41.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.621+0000 7f17a37fe700 1 -- 172.21.15.78:0/2230679542 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f17b001c8e0 con 0x7f17bc0f5680 2023-12-21T03:54:41.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.621+0000 7f17a37fe700 1 --2- 172.21.15.78:0/2230679542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f17a405c300 0x7f17a405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:41.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.622+0000 7f17a37fe700 1 -- 172.21.15.78:0/2230679542 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f17b0077b00 con 0x7f17bc0f5680 2023-12-21T03:54:41.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.622+0000 7f17ba7fc700 1 --2- 172.21.15.78:0/2230679542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f17a405c300 0x7f17a405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:41.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.622+0000 7f17ba7fc700 1 --2- 172.21.15.78:0/2230679542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f17a405c300 0x7f17a405e7c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f17ac00b790 tx=0x7f17ac008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:41.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.625+0000 7f17a37fe700 1 -- 172.21.15.78:0/2230679542 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17b004aa20 con 0x7f17bc0f5680 2023-12-21T03:54:41.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:39.667264+0000 mgr.smithi078.zidbgq (mgr.14162) 512 : cluster [DBG] pgmap v480: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:41.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:39.667264+0000 mgr.smithi078.zidbgq (mgr.14162) 512 : cluster [DBG] pgmap v480: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.837+0000 7f17c137b700 1 -- 172.21.15.78:0/2230679542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f17bc0fe050 con 0x7f17bc0f5680 2023-12-21T03:54:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.838+0000 7f17a37fe700 1 -- 172.21.15.78:0/2230679542 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f17bc0fe050 con 0x7f17bc0f5680 2023-12-21T03:54:41.839 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:41.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.839+0000 7f17c137b700 1 -- 172.21.15.78:0/2230679542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f17a405c300 msgr2=0x7f17a405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.839+0000 7f17c137b700 1 --2- 172.21.15.78:0/2230679542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f17a405c300 0x7f17a405e7c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f17ac00b790 tx=0x7f17ac008040).stop 2023-12-21T03:54:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.839+0000 7f17c137b700 1 -- 172.21.15.78:0/2230679542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17bc0f5680 msgr2=0x7f17bc109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.839+0000 7f17c137b700 1 --2- 172.21.15.78:0/2230679542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17bc0f5680 0x7f17bc109cd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f17b0009830 tx=0x7f17b0009b30).stop 2023-12-21T03:54:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.839+0000 7f17c137b700 1 -- 172.21.15.78:0/2230679542 shutdown_connections 2023-12-21T03:54:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.839+0000 7f17c137b700 1 --2- 172.21.15.78:0/2230679542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f17a405c300 0x7f17a405e7c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.839+0000 7f17c137b700 1 --2- 172.21.15.78:0/2230679542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17bc0f5680 0x7f17bc109cd0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.839+0000 7f17c137b700 1 --2- 172.21.15.78:0/2230679542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17bc0f5f90 0x7f17bc10c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.839+0000 7f17c137b700 1 -- 172.21.15.78:0/2230679542 >> 172.21.15.78:0/2230679542 conn(0x7f17bc0f0e80 msgr2=0x7f17bc0fa410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.839+0000 7f17c137b700 1 -- 172.21.15.78:0/2230679542 shutdown_connections 2023-12-21T03:54:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:41.839+0000 7f17c137b700 1 -- 172.21.15.78:0/2230679542 wait complete. 2023-12-21T03:54:42.213 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:42.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:42 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:41.839128+0000 mon.smithi078 (mon.0) 540 : audit [DBG] from='client.? 172.21.15.78:0/2230679542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:42.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:42 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:41.839128+0000 mon.smithi078 (mon.0) 540 : audit [DBG] from='client.? 172.21.15.78:0/2230679542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:41.667611+0000 mgr.smithi078.zidbgq (mgr.14162) 513 : cluster [DBG] pgmap v481: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:41.667611+0000 mgr.smithi078.zidbgq (mgr.14162) 513 : cluster [DBG] pgmap v481: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:45.213 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:45.558 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:43.668029+0000 mgr.smithi078.zidbgq (mgr.14162) 514 : cluster [DBG] pgmap v482: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:43.668029+0000 mgr.smithi078.zidbgq (mgr.14162) 514 : cluster [DBG] pgmap v482: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:46.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.388+0000 7f6016a88700 1 -- 172.21.15.78:0/2404728568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60100f67c0 msgr2=0x7f60100f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.388+0000 7f6016a88700 1 --2- 172.21.15.78:0/2404728568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60100f67c0 0x7f60100f6be0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f6004004660 tx=0x7f6004009b30).stop 2023-12-21T03:54:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.388+0000 7f6016a88700 1 -- 172.21.15.78:0/2404728568 shutdown_connections 2023-12-21T03:54:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.388+0000 7f6016a88700 1 --2- 172.21.15.78:0/2404728568 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60100f7a60 0x7f60100f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.388+0000 7f6016a88700 1 --2- 172.21.15.78:0/2404728568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60100f67c0 0x7f60100f6be0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.388+0000 7f6016a88700 1 -- 172.21.15.78:0/2404728568 >> 172.21.15.78:0/2404728568 conn(0x7f60100f1f20 msgr2=0x7f60100f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.388+0000 7f6016a88700 1 -- 172.21.15.78:0/2404728568 shutdown_connections 2023-12-21T03:54:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.388+0000 7f6016a88700 1 -- 172.21.15.78:0/2404728568 wait complete. 2023-12-21T03:54:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.388+0000 7f6016a88700 1 Processor -- start 2023-12-21T03:54:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.388+0000 7f6016a88700 1 -- start start 2023-12-21T03:54:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6016a88700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60100f67c0 0x7f601006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6016a88700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60100f7a60 0x7f6010069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6016a88700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601006a380 con 0x7f60100f67c0 2023-12-21T03:54:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6016a88700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601006a4c0 con 0x7f60100f7a60 2023-12-21T03:54:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6014824700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60100f67c0 0x7f601006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6014824700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60100f67c0 0x7f601006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47610/0 (socket says 172.21.15.78:47610) 2023-12-21T03:54:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6014824700 1 -- 172.21.15.78:0/270590701 learned_addr learned my addr 172.21.15.78:0/270590701 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f600ffff700 1 --2- 172.21.15.78:0/270590701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60100f7a60 0x7f6010069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6014824700 1 -- 172.21.15.78:0/270590701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60100f7a60 msgr2=0x7f6010069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6014824700 1 --2- 172.21.15.78:0/270590701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60100f7a60 0x7f6010069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6014824700 1 -- 172.21.15.78:0/270590701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6004005040 con 0x7f60100f67c0 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6014824700 1 --2- 172.21.15.78:0/270590701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60100f67c0 0x7f601006b9a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f6004004e30 tx=0x7f6004009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f600d7fa700 1 -- 172.21.15.78:0/270590701 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6004018070 con 0x7f60100f67c0 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f6016a88700 1 -- 172.21.15.78:0/270590701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601006a740 con 0x7f60100f67c0 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f600d7fa700 1 -- 172.21.15.78:0/270590701 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6004013410 con 0x7f60100f67c0 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.389+0000 7f600d7fa700 1 -- 172.21.15.78:0/270590701 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f600401de20 con 0x7f60100f67c0 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.390+0000 7f6016a88700 1 -- 172.21.15.78:0/270590701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601006aca0 con 0x7f60100f67c0 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.391+0000 7f6016a88700 1 -- 172.21.15.78:0/270590701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6010045ad0 con 0x7f60100f67c0 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.391+0000 7f600d7fa700 1 -- 172.21.15.78:0/270590701 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f600401d420 con 0x7f60100f67c0 2023-12-21T03:54:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.391+0000 7f600d7fa700 1 --2- 172.21.15.78:0/270590701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f600005c2a0 0x7f600005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:46.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.391+0000 7f600d7fa700 1 -- 172.21.15.78:0/270590701 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f60040777b0 con 0x7f60100f67c0 2023-12-21T03:54:46.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.391+0000 7f600ffff700 1 --2- 172.21.15.78:0/270590701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f600005c2a0 0x7f600005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:46.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.392+0000 7f600ffff700 1 --2- 172.21.15.78:0/270590701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f600005c2a0 0x7f600005e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5ffc004e20 tx=0x7f5ffc00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.395+0000 7f600d7fa700 1 -- 172.21.15.78:0/270590701 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f600404a7b0 con 0x7f60100f67c0 2023-12-21T03:54:46.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.608+0000 7f6016a88700 1 -- 172.21.15.78:0/270590701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6010027a00 con 0x7f60100f67c0 2023-12-21T03:54:46.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.609+0000 7f600d7fa700 1 -- 172.21.15.78:0/270590701 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f600404ddd0 con 0x7f60100f67c0 2023-12-21T03:54:46.610 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:46.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.610+0000 7f6016a88700 1 -- 172.21.15.78:0/270590701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f600005c2a0 msgr2=0x7f600005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:46.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.610+0000 7f6016a88700 1 --2- 172.21.15.78:0/270590701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f600005c2a0 0x7f600005e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5ffc004e20 tx=0x7f5ffc00b040).stop 2023-12-21T03:54:46.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.610+0000 7f6016a88700 1 -- 172.21.15.78:0/270590701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60100f67c0 msgr2=0x7f601006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:46.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.610+0000 7f6016a88700 1 --2- 172.21.15.78:0/270590701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60100f67c0 0x7f601006b9a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f6004004e30 tx=0x7f6004009b30).stop 2023-12-21T03:54:46.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.610+0000 7f6016a88700 1 -- 172.21.15.78:0/270590701 shutdown_connections 2023-12-21T03:54:46.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.610+0000 7f6016a88700 1 --2- 172.21.15.78:0/270590701 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f600005c2a0 0x7f600005e760 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:46.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.610+0000 7f6016a88700 1 --2- 172.21.15.78:0/270590701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60100f67c0 0x7f601006b9a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:46.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.610+0000 7f6016a88700 1 --2- 172.21.15.78:0/270590701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60100f7a60 0x7f6010069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:46.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.610+0000 7f6016a88700 1 -- 172.21.15.78:0/270590701 >> 172.21.15.78:0/270590701 conn(0x7f60100f1f20 msgr2=0x7f60100fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:46.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.610+0000 7f6016a88700 1 -- 172.21.15.78:0/270590701 shutdown_connections 2023-12-21T03:54:46.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:46.610+0000 7f6016a88700 1 -- 172.21.15.78:0/270590701 wait complete. 2023-12-21T03:54:46.996 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:45.668349+0000 mgr.smithi078.zidbgq (mgr.14162) 515 : cluster [DBG] pgmap v483: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:46.610064+0000 mon.smithi078 (mon.0) 541 : audit [DBG] from='client.? 172.21.15.78:0/270590701' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:45.668349+0000 mgr.smithi078.zidbgq (mgr.14162) 515 : cluster [DBG] pgmap v483: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:46.610064+0000 mon.smithi078 (mon.0) 541 : audit [DBG] from='client.? 172.21.15.78:0/270590701' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:47.668798+0000 mgr.smithi078.zidbgq (mgr.14162) 516 : cluster [DBG] pgmap v484: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:49.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:47.668798+0000 mgr.smithi078.zidbgq (mgr.14162) 516 : cluster [DBG] pgmap v484: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:49.997 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:50.235 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.200+0000 7fd9764e2700 1 -- 172.21.15.78:0/1386980807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9700f67b0 msgr2=0x7fd9700f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.200+0000 7fd9764e2700 1 --2- 172.21.15.78:0/1386980807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9700f67b0 0x7fd9700f6c10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fd960004660 tx=0x7fd960009b30).stop 2023-12-21T03:54:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.200+0000 7fd9764e2700 1 -- 172.21.15.78:0/1386980807 shutdown_connections 2023-12-21T03:54:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.200+0000 7fd9764e2700 1 --2- 172.21.15.78:0/1386980807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9700f67b0 0x7fd9700f6c10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.200+0000 7fd9764e2700 1 --2- 172.21.15.78:0/1386980807 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9700f5510 0x7fd9700f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.200+0000 7fd9764e2700 1 -- 172.21.15.78:0/1386980807 >> 172.21.15.78:0/1386980807 conn(0x7fd9700f0c70 msgr2=0x7fd9700f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.201+0000 7fd9764e2700 1 -- 172.21.15.78:0/1386980807 shutdown_connections 2023-12-21T03:54:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.201+0000 7fd9764e2700 1 -- 172.21.15.78:0/1386980807 wait complete. 2023-12-21T03:54:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.201+0000 7fd9764e2700 1 Processor -- start 2023-12-21T03:54:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.201+0000 7fd9764e2700 1 -- start start 2023-12-21T03:54:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.201+0000 7fd9764e2700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9700f5510 0x7fd970109b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.201+0000 7fd9764e2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9700f67b0 0x7fd97010c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.201+0000 7fd9764e2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd970060980 con 0x7fd9700f67b0 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.201+0000 7fd9764e2700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd970060ac0 con 0x7fd9700f5510 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.201+0000 7fd974cdf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9700f67b0 0x7fd97010c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.201+0000 7fd974cdf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9700f67b0 0x7fd97010c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47830/0 (socket says 172.21.15.78:47830) 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.201+0000 7fd974cdf700 1 -- 172.21.15.78:0/563338537 learned_addr learned my addr 172.21.15.78:0/563338537 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.202+0000 7fd9754e0700 1 --2- 172.21.15.78:0/563338537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9700f5510 0x7fd970109b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.202+0000 7fd974cdf700 1 -- 172.21.15.78:0/563338537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9700f5510 msgr2=0x7fd970109b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.202+0000 7fd974cdf700 1 --2- 172.21.15.78:0/563338537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9700f5510 0x7fd970109b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.202+0000 7fd974cdf700 1 -- 172.21.15.78:0/563338537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd960005040 con 0x7fd9700f67b0 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.202+0000 7fd974cdf700 1 --2- 172.21.15.78:0/563338537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9700f67b0 0x7fd97010c090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fd960009830 tx=0x7fd960009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.202+0000 7fd965ffb700 1 -- 172.21.15.78:0/563338537 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd960018070 con 0x7fd9700f67b0 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.202+0000 7fd965ffb700 1 -- 172.21.15.78:0/563338537 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd960013410 con 0x7fd9700f67b0 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.202+0000 7fd9764e2700 1 -- 172.21.15.78:0/563338537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97010c640 con 0x7fd9700f67b0 2023-12-21T03:54:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.202+0000 7fd965ffb700 1 -- 172.21.15.78:0/563338537 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd96001c620 con 0x7fd9700f67b0 2023-12-21T03:54:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.202+0000 7fd9764e2700 1 -- 172.21.15.78:0/563338537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97010cb70 con 0x7fd9700f67b0 2023-12-21T03:54:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.203+0000 7fd9764e2700 1 -- 172.21.15.78:0/563338537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd970059100 con 0x7fd9700f67b0 2023-12-21T03:54:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.203+0000 7fd965ffb700 1 -- 172.21.15.78:0/563338537 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd96001d420 con 0x7fd9700f67b0 2023-12-21T03:54:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.204+0000 7fd965ffb700 1 --2- 172.21.15.78:0/563338537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd95c05c2b0 0x7fd95c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.204+0000 7fd965ffb700 1 -- 172.21.15.78:0/563338537 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd9600779d0 con 0x7fd9700f67b0 2023-12-21T03:54:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.204+0000 7fd9754e0700 1 --2- 172.21.15.78:0/563338537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd95c05c2b0 0x7fd95c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.204+0000 7fd9754e0700 1 --2- 172.21.15.78:0/563338537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd95c05c2b0 0x7fd95c05e770 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fd970060600 tx=0x7fd96c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:51.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.208+0000 7fd965ffb700 1 -- 172.21.15.78:0/563338537 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd96004aa50 con 0x7fd9700f67b0 2023-12-21T03:54:51.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.416+0000 7fd9764e2700 1 -- 172.21.15.78:0/563338537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd970045ad0 con 0x7fd9700f67b0 2023-12-21T03:54:51.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.416+0000 7fd965ffb700 1 -- 172.21.15.78:0/563338537 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fd96004e070 con 0x7fd9700f67b0 2023-12-21T03:54:51.418 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.418+0000 7fd9764e2700 1 -- 172.21.15.78:0/563338537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd95c05c2b0 msgr2=0x7fd95c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.418+0000 7fd9764e2700 1 --2- 172.21.15.78:0/563338537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd95c05c2b0 0x7fd95c05e770 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fd970060600 tx=0x7fd96c009040).stop 2023-12-21T03:54:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.418+0000 7fd9764e2700 1 -- 172.21.15.78:0/563338537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9700f67b0 msgr2=0x7fd97010c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.418+0000 7fd9764e2700 1 --2- 172.21.15.78:0/563338537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9700f67b0 0x7fd97010c090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fd960009830 tx=0x7fd960009b30).stop 2023-12-21T03:54:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.418+0000 7fd9764e2700 1 -- 172.21.15.78:0/563338537 shutdown_connections 2023-12-21T03:54:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.418+0000 7fd9764e2700 1 --2- 172.21.15.78:0/563338537 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd95c05c2b0 0x7fd95c05e770 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.418+0000 7fd9764e2700 1 --2- 172.21.15.78:0/563338537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9700f5510 0x7fd970109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.418+0000 7fd9764e2700 1 --2- 172.21.15.78:0/563338537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9700f67b0 0x7fd97010c090 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.418+0000 7fd9764e2700 1 -- 172.21.15.78:0/563338537 >> 172.21.15.78:0/563338537 conn(0x7fd9700f0c70 msgr2=0x7fd9700fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.418+0000 7fd9764e2700 1 -- 172.21.15.78:0/563338537 shutdown_connections 2023-12-21T03:54:51.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:51.418+0000 7fd9764e2700 1 -- 172.21.15.78:0/563338537 wait complete. 2023-12-21T03:54:51.736 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:49.669337+0000 mgr.smithi078.zidbgq (mgr.14162) 517 : cluster [DBG] pgmap v485: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:51.736 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:51.417629+0000 mon.smithi078 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.78:0/563338537' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:49.669337+0000 mgr.smithi078.zidbgq (mgr.14162) 517 : cluster [DBG] pgmap v485: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:51.417629+0000 mon.smithi078 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.78:0/563338537' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:51.811 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:53.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:51.669801+0000 mgr.smithi078.zidbgq (mgr.14162) 518 : cluster [DBG] pgmap v486: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:51.669801+0000 mgr.smithi078.zidbgq (mgr.14162) 518 : cluster [DBG] pgmap v486: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:54.812 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:55.045 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:54:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:53.670272+0000 mgr.smithi078.zidbgq (mgr.14162) 519 : cluster [DBG] pgmap v487: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:53.670272+0000 mgr.smithi078.zidbgq (mgr.14162) 519 : cluster [DBG] pgmap v487: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:56.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.053+0000 7f68b2f76700 1 -- 172.21.15.78:0/1883030354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68ac0f56e0 msgr2=0x7f68ac0f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:56.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.053+0000 7f68b2f76700 1 --2- 172.21.15.78:0/1883030354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68ac0f56e0 0x7f68ac0f5b00 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f68a0004660 tx=0x7f68a0009b30).stop 2023-12-21T03:54:56.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.053+0000 7f68b2f76700 1 -- 172.21.15.78:0/1883030354 shutdown_connections 2023-12-21T03:54:56.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.053+0000 7f68b2f76700 1 --2- 172.21.15.78:0/1883030354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68ac0f6980 0x7f68ac0f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:54:56.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.053+0000 7f68b2f76700 1 --2- 172.21.15.78:0/1883030354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68ac0f56e0 0x7f68ac0f5b00 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:56.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.053+0000 7f68b2f76700 1 -- 172.21.15.78:0/1883030354 >> 172.21.15.78:0/1883030354 conn(0x7f68ac0f0e20 msgr2=0x7f68ac0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:56.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.053+0000 7f68b2f76700 1 -- 172.21.15.78:0/1883030354 shutdown_connections 2023-12-21T03:54:56.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.053+0000 7f68b2f76700 1 -- 172.21.15.78:0/1883030354 wait complete. 2023-12-21T03:54:56.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.053+0000 7f68b2f76700 1 Processor -- start 2023-12-21T03:54:56.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.053+0000 7f68b2f76700 1 -- start start 2023-12-21T03:54:56.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68b2f76700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68ac0f56e0 0x7f68ac0f9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:56.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68b2f76700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68ac0f6980 0x7f68ac0f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:56.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68b2f76700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0f82b0 con 0x7f68ac0f6980 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68b2f76700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0f83f0 con 0x7f68ac0f56e0 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68abfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68ac0f6980 0x7f68ac0f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68abfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68ac0f6980 0x7f68ac0f7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47852/0 (socket says 172.21.15.78:47852) 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68abfff700 1 -- 172.21.15.78:0/151143143 learned_addr learned my addr 172.21.15.78:0/151143143 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68b0d12700 1 --2- 172.21.15.78:0/151143143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68ac0f56e0 0x7f68ac0f9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68abfff700 1 -- 172.21.15.78:0/151143143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68ac0f56e0 msgr2=0x7f68ac0f9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68abfff700 1 --2- 172.21.15.78:0/151143143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68ac0f56e0 0x7f68ac0f9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68abfff700 1 -- 172.21.15.78:0/151143143 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a0005040 con 0x7f68ac0f6980 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68abfff700 1 --2- 172.21.15.78:0/151143143 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68ac0f6980 0x7f68ac0f7cc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f689800b410 tx=0x7f6898006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68a97fa700 1 -- 172.21.15.78:0/151143143 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68980125f0 con 0x7f68ac0f6980 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68b2f76700 1 -- 172.21.15.78:0/151143143 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac0f86d0 con 0x7f68ac0f6980 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68a97fa700 1 -- 172.21.15.78:0/151143143 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6898012c00 con 0x7f68ac0f6980 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68a97fa700 1 -- 172.21.15.78:0/151143143 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6898002aa0 con 0x7f68ac0f6980 2023-12-21T03:54:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.054+0000 7f68b2f76700 1 -- 172.21.15.78:0/151143143 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac10f0f0 con 0x7f68ac0f6980 2023-12-21T03:54:56.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.055+0000 7f68b2f76700 1 -- 172.21.15.78:0/151143143 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ac045ad0 con 0x7f68ac0f6980 2023-12-21T03:54:56.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.055+0000 7f68a97fa700 1 -- 172.21.15.78:0/151143143 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f689800f030 con 0x7f68ac0f6980 2023-12-21T03:54:56.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.056+0000 7f68a97fa700 1 --2- 172.21.15.78:0/151143143 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f689c05c2a0 0x7f689c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:54:56.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.056+0000 7f68a97fa700 1 -- 172.21.15.78:0/151143143 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f6898076060 con 0x7f68ac0f6980 2023-12-21T03:54:56.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.056+0000 7f68b0d12700 1 --2- 172.21.15.78:0/151143143 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f689c05c2a0 0x7f689c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:54:56.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.056+0000 7f68b0d12700 1 --2- 172.21.15.78:0/151143143 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f689c05c2a0 0x7f689c05e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f68ac0f95d0 tx=0x7f68a0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:54:56.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.060+0000 7f68a97fa700 1 -- 172.21.15.78:0/151143143 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6898049000 con 0x7f68ac0f6980 2023-12-21T03:54:56.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.267+0000 7f68b2f76700 1 -- 172.21.15.78:0/151143143 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f68ac0fdef0 con 0x7f68ac0f6980 2023-12-21T03:54:56.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.268+0000 7f68a97fa700 1 -- 172.21.15.78:0/151143143 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f689804c620 con 0x7f68ac0f6980 2023-12-21T03:54:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:54:56.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.269+0000 7f68b2f76700 1 -- 172.21.15.78:0/151143143 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f689c05c2a0 msgr2=0x7f689c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:56.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.269+0000 7f68b2f76700 1 --2- 172.21.15.78:0/151143143 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f689c05c2a0 0x7f689c05e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f68ac0f95d0 tx=0x7f68a0009b30).stop 2023-12-21T03:54:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.269+0000 7f68b2f76700 1 -- 172.21.15.78:0/151143143 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68ac0f6980 msgr2=0x7f68ac0f7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:54:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.269+0000 7f68b2f76700 1 --2- 172.21.15.78:0/151143143 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68ac0f6980 0x7f68ac0f7cc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f689800b410 tx=0x7f6898006ef0).stop 2023-12-21T03:54:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.269+0000 7f68b2f76700 1 -- 172.21.15.78:0/151143143 shutdown_connections 2023-12-21T03:54:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.269+0000 7f68b2f76700 1 --2- 172.21.15.78:0/151143143 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f689c05c2a0 0x7f689c05e760 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.269+0000 7f68b2f76700 1 --2- 172.21.15.78:0/151143143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68ac0f56e0 0x7f68ac0f9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.269+0000 7f68b2f76700 1 --2- 172.21.15.78:0/151143143 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68ac0f6980 0x7f68ac0f7cc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:54:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.269+0000 7f68b2f76700 1 -- 172.21.15.78:0/151143143 >> 172.21.15.78:0/151143143 conn(0x7f68ac0f0e20 msgr2=0x7f68ac0603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:54:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.269+0000 7f68b2f76700 1 -- 172.21.15.78:0/151143143 shutdown_connections 2023-12-21T03:54:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:54:56.269+0000 7f68b2f76700 1 -- 172.21.15.78:0/151143143 wait complete. 2023-12-21T03:54:56.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:56 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:54:56.269016+0000 mon.smithi078 (mon.0) 543 : audit [DBG] from='client.? 172.21.15.78:0/151143143' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:56.685 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:54:56.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:56 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:54:56.269016+0000 mon.smithi078 (mon.0) 543 : audit [DBG] from='client.? 172.21.15.78:0/151143143' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:54:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:55.670578+0000 mgr.smithi078.zidbgq (mgr.14162) 520 : cluster [DBG] pgmap v488: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:55.670578+0000 mgr.smithi078.zidbgq (mgr.14162) 520 : cluster [DBG] pgmap v488: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:59.686 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:54:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:54:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:57.670955+0000 mgr.smithi078.zidbgq (mgr.14162) 521 : cluster [DBG] pgmap v489: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:59.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:54:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:57.670955+0000 mgr.smithi078.zidbgq (mgr.14162) 521 : cluster [DBG] pgmap v489: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:54:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:00.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.916+0000 7fbcd247f700 1 -- 172.21.15.78:0/1299106412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbccc0f9980 msgr2=0x7fbccc0fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:00.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.916+0000 7fbcd247f700 1 --2- 172.21.15.78:0/1299106412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbccc0f9980 0x7fbccc0fbd70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fbcc0004660 tx=0x7fbcc0009b30).stop 2023-12-21T03:55:00.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.916+0000 7fbcd247f700 1 -- 172.21.15.78:0/1299106412 shutdown_connections 2023-12-21T03:55:00.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.916+0000 7fbcd247f700 1 --2- 172.21.15.78:0/1299106412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbccc0f9980 0x7fbccc0fbd70 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:00.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.916+0000 7fbcd247f700 1 --2- 172.21.15.78:0/1299106412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc060810 0x7fbccc0f9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:00.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.916+0000 7fbcd247f700 1 -- 172.21.15.78:0/1299106412 >> 172.21.15.78:0/1299106412 conn(0x7fbccc0f0e20 msgr2=0x7fbccc0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:00.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.916+0000 7fbcd247f700 1 -- 172.21.15.78:0/1299106412 shutdown_connections 2023-12-21T03:55:00.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.916+0000 7fbcd247f700 1 -- 172.21.15.78:0/1299106412 wait complete. 2023-12-21T03:55:00.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.917+0000 7fbcd247f700 1 Processor -- start 2023-12-21T03:55:00.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.917+0000 7fbcd247f700 1 -- start start 2023-12-21T03:55:00.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.917+0000 7fbcd247f700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc060810 0x7fbccc1070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:00.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.917+0000 7fbcd247f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbccc0f9980 0x7fbccc107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:00.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.917+0000 7fbcd247f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc107b40 con 0x7fbccc0f9980 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.917+0000 7fbcd247f700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc107c80 con 0x7fbccc060810 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.917+0000 7fbccb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbccc0f9980 0x7fbccc107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.917+0000 7fbccb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbccc0f9980 0x7fbccc107590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59376/0 (socket says 172.21.15.78:59376) 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.917+0000 7fbccb7fe700 1 -- 172.21.15.78:0/3213289789 learned_addr learned my addr 172.21.15.78:0/3213289789 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.917+0000 7fbccbfff700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc060810 0x7fbccc1070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.918+0000 7fbccb7fe700 1 -- 172.21.15.78:0/3213289789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc060810 msgr2=0x7fbccc1070a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.918+0000 7fbccb7fe700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc060810 0x7fbccc1070a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.918+0000 7fbccb7fe700 1 -- 172.21.15.78:0/3213289789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc0005040 con 0x7fbccc0f9980 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.918+0000 7fbccbfff700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc060810 0x7fbccc1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.918+0000 7fbccb7fe700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbccc0f9980 0x7fbccc107590 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fbcc0009f50 tx=0x7fbcc0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.918+0000 7fbcc8ff9700 1 -- 172.21.15.78:0/3213289789 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcc0018070 con 0x7fbccc0f9980 2023-12-21T03:55:00.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.918+0000 7fbcc8ff9700 1 -- 172.21.15.78:0/3213289789 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcc0013410 con 0x7fbccc0f9980 2023-12-21T03:55:00.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.918+0000 7fbcd247f700 1 -- 172.21.15.78:0/3213289789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc1085c0 con 0x7fbccc0f9980 2023-12-21T03:55:00.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.918+0000 7fbcc8ff9700 1 -- 172.21.15.78:0/3213289789 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcc001c620 con 0x7fbccc0f9980 2023-12-21T03:55:00.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.918+0000 7fbcd247f700 1 -- 172.21.15.78:0/3213289789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc108b20 con 0x7fbccc0f9980 2023-12-21T03:55:00.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.919+0000 7fbcd247f700 1 -- 172.21.15.78:0/3213289789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbccc045ad0 con 0x7fbccc0f9980 2023-12-21T03:55:00.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.919+0000 7fbcc8ff9700 1 -- 172.21.15.78:0/3213289789 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbcc001d420 con 0x7fbccc0f9980 2023-12-21T03:55:00.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.920+0000 7fbcc8ff9700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbcb405c2a0 0x7fbcb405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:00.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.920+0000 7fbcc8ff9700 1 -- 172.21.15.78:0/3213289789 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbcc0077a00 con 0x7fbccc0f9980 2023-12-21T03:55:00.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.920+0000 7fbccbfff700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbcb405c2a0 0x7fbcb405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:00.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.920+0000 7fbccbfff700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbcb405c2a0 0x7fbcb405e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fbcbc004a90 tx=0x7fbcbc009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:00.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:00.924+0000 7fbcc8ff9700 1 -- 172.21.15.78:0/3213289789 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcc004aa00 con 0x7fbccc0f9980 2023-12-21T03:55:01.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.130+0000 7fbcd247f700 1 -- 172.21.15.78:0/3213289789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbccc027a00 con 0x7fbccc0f9980 2023-12-21T03:55:01.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.130+0000 7fbcc8ff9700 1 -- 172.21.15.78:0/3213289789 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fbcc004e020 con 0x7fbccc0f9980 2023-12-21T03:55:01.131 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:01.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.132+0000 7fbcd247f700 1 -- 172.21.15.78:0/3213289789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbcb405c2a0 msgr2=0x7fbcb405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:01.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.132+0000 7fbcd247f700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbcb405c2a0 0x7fbcb405e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fbcbc004a90 tx=0x7fbcbc009040).stop 2023-12-21T03:55:01.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.132+0000 7fbcd247f700 1 -- 172.21.15.78:0/3213289789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbccc0f9980 msgr2=0x7fbccc107590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:01.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.132+0000 7fbcd247f700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbccc0f9980 0x7fbccc107590 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fbcc0009f50 tx=0x7fbcc0009b30).stop 2023-12-21T03:55:01.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.132+0000 7fbcd247f700 1 -- 172.21.15.78:0/3213289789 shutdown_connections 2023-12-21T03:55:01.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.132+0000 7fbcd247f700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fbcb405c2a0 0x7fbcb405e760 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:01.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.132+0000 7fbcd247f700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc060810 0x7fbccc1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:01.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.132+0000 7fbcd247f700 1 --2- 172.21.15.78:0/3213289789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbccc0f9980 0x7fbccc107590 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:01.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.132+0000 7fbcd247f700 1 -- 172.21.15.78:0/3213289789 >> 172.21.15.78:0/3213289789 conn(0x7fbccc0f0e20 msgr2=0x7fbccc0f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:01.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.132+0000 7fbcd247f700 1 -- 172.21.15.78:0/3213289789 shutdown_connections 2023-12-21T03:55:01.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:01.132+0000 7fbcd247f700 1 -- 172.21.15.78:0/3213289789 wait complete. 2023-12-21T03:55:01.494 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:54:59.671303+0000 mgr.smithi078.zidbgq (mgr.14162) 522 : cluster [DBG] pgmap v490: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:01.131434+0000 mon.smithi078 (mon.0) 544 : audit [DBG] from='client.? 172.21.15.78:0/3213289789' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:54:59.671303+0000 mgr.smithi078.zidbgq (mgr.14162) 522 : cluster [DBG] pgmap v490: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:01.131434+0000 mon.smithi078 (mon.0) 544 : audit [DBG] from='client.? 172.21.15.78:0/3213289789' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:03.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:01.671597+0000 mgr.smithi078.zidbgq (mgr.14162) 523 : cluster [DBG] pgmap v491: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:01.671597+0000 mgr.smithi078.zidbgq (mgr.14162) 523 : cluster [DBG] pgmap v491: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:04.495 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.683+0000 7fb06084e700 1 -- 172.21.15.78:0/3381604152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb05c0f56c0 msgr2=0x7fb05c0f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.683+0000 7fb06084e700 1 --2- 172.21.15.78:0/3381604152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb05c0f56c0 0x7fb05c0f5ae0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fb04c004660 tx=0x7fb04c009b30).stop 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.683+0000 7fb06084e700 1 -- 172.21.15.78:0/3381604152 shutdown_connections 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.683+0000 7fb06084e700 1 --2- 172.21.15.78:0/3381604152 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb05c0f6960 0x7fb05c0f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.683+0000 7fb06084e700 1 --2- 172.21.15.78:0/3381604152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb05c0f56c0 0x7fb05c0f5ae0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.683+0000 7fb06084e700 1 -- 172.21.15.78:0/3381604152 >> 172.21.15.78:0/3381604152 conn(0x7fb05c0f0e20 msgr2=0x7fb05c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.683+0000 7fb06084e700 1 -- 172.21.15.78:0/3381604152 shutdown_connections 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.683+0000 7fb06084e700 1 -- 172.21.15.78:0/3381604152 wait complete. 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb06084e700 1 Processor -- start 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb06084e700 1 -- start start 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb06084e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb05c0f56c0 0x7fb05c1070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb06084e700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb05c0f6960 0x7fb05c1075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb06084e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c107c10 con 0x7fb05c0f56c0 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb06084e700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c1084c0 con 0x7fb05c0f6960 2023-12-21T03:55:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb05a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb05c0f56c0 0x7fb05c1070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb05a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb05c0f56c0 0x7fb05c1070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59386/0 (socket says 172.21.15.78:59386) 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb05a59c700 1 -- 172.21.15.78:0/3627860789 learned_addr learned my addr 172.21.15.78:0/3627860789 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb059d9b700 1 --2- 172.21.15.78:0/3627860789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb05c0f6960 0x7fb05c1075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb05a59c700 1 -- 172.21.15.78:0/3627860789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb05c0f6960 msgr2=0x7fb05c1075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb05a59c700 1 --2- 172.21.15.78:0/3627860789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb05c0f6960 0x7fb05c1075d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb05a59c700 1 -- 172.21.15.78:0/3627860789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c005040 con 0x7fb05c0f56c0 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.684+0000 7fb05a59c700 1 --2- 172.21.15.78:0/3627860789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb05c0f56c0 0x7fb05c1070e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fb04c009f50 tx=0x7fb04c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.685+0000 7fb04affd700 1 -- 172.21.15.78:0/3627860789 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb04c018070 con 0x7fb05c0f56c0 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.685+0000 7fb04affd700 1 -- 172.21.15.78:0/3627860789 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb04c013410 con 0x7fb05c0f56c0 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.685+0000 7fb06084e700 1 -- 172.21.15.78:0/3627860789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05c108740 con 0x7fb05c0f56c0 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.685+0000 7fb04affd700 1 -- 172.21.15.78:0/3627860789 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb04c01c620 con 0x7fb05c0f56c0 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.685+0000 7fb06084e700 1 -- 172.21.15.78:0/3627860789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05c108ca0 con 0x7fb05c0f56c0 2023-12-21T03:55:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.686+0000 7fb06084e700 1 -- 172.21.15.78:0/3627860789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05c059100 con 0x7fb05c0f56c0 2023-12-21T03:55:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.686+0000 7fb04affd700 1 -- 172.21.15.78:0/3627860789 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb04c01d420 con 0x7fb05c0f56c0 2023-12-21T03:55:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.686+0000 7fb04affd700 1 --2- 172.21.15.78:0/3627860789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb04405c2a0 0x7fb04405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.686+0000 7fb04affd700 1 -- 172.21.15.78:0/3627860789 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb04c077a40 con 0x7fb05c0f56c0 2023-12-21T03:55:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.686+0000 7fb059d9b700 1 --2- 172.21.15.78:0/3627860789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb04405c2a0 0x7fb04405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.687+0000 7fb059d9b700 1 --2- 172.21.15.78:0/3627860789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb04405c2a0 0x7fb04405e760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fb05000b790 tx=0x7fb050008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.690+0000 7fb04affd700 1 -- 172.21.15.78:0/3627860789 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb04c04aa40 con 0x7fb05c0f56c0 2023-12-21T03:55:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:03.672065+0000 mgr.smithi078.zidbgq (mgr.14162) 524 : cluster [DBG] pgmap v492: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:05.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:03.672065+0000 mgr.smithi078.zidbgq (mgr.14162) 524 : cluster [DBG] pgmap v492: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:05.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.899+0000 7fb06084e700 1 -- 172.21.15.78:0/3627860789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb05c0fe050 con 0x7fb05c0f56c0 2023-12-21T03:55:05.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.899+0000 7fb04affd700 1 -- 172.21.15.78:0/3627860789 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fb04c04e060 con 0x7fb05c0f56c0 2023-12-21T03:55:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:05.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.901+0000 7fb06084e700 1 -- 172.21.15.78:0/3627860789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb04405c2a0 msgr2=0x7fb04405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:05.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.901+0000 7fb06084e700 1 --2- 172.21.15.78:0/3627860789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb04405c2a0 0x7fb04405e760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fb05000b790 tx=0x7fb050008040).stop 2023-12-21T03:55:05.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.901+0000 7fb06084e700 1 -- 172.21.15.78:0/3627860789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb05c0f56c0 msgr2=0x7fb05c1070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:05.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.901+0000 7fb06084e700 1 --2- 172.21.15.78:0/3627860789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb05c0f56c0 0x7fb05c1070e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fb04c009f50 tx=0x7fb04c009b30).stop 2023-12-21T03:55:05.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.901+0000 7fb06084e700 1 -- 172.21.15.78:0/3627860789 shutdown_connections 2023-12-21T03:55:05.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.901+0000 7fb06084e700 1 --2- 172.21.15.78:0/3627860789 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb04405c2a0 0x7fb04405e760 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:05.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.901+0000 7fb06084e700 1 --2- 172.21.15.78:0/3627860789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb05c0f56c0 0x7fb05c1070e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:05.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.901+0000 7fb06084e700 1 --2- 172.21.15.78:0/3627860789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb05c0f6960 0x7fb05c1075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:05.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.901+0000 7fb06084e700 1 -- 172.21.15.78:0/3627860789 >> 172.21.15.78:0/3627860789 conn(0x7fb05c0f0e20 msgr2=0x7fb05c0603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:05.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.901+0000 7fb06084e700 1 -- 172.21.15.78:0/3627860789 shutdown_connections 2023-12-21T03:55:05.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:05.901+0000 7fb06084e700 1 -- 172.21.15.78:0/3627860789 wait complete. 2023-12-21T03:55:06.275 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:06.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:06 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:05.900764+0000 mon.smithi078 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.78:0/3627860789' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:07.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:06 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:05.900764+0000 mon.smithi078 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.78:0/3627860789' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:05.672414+0000 mgr.smithi078.zidbgq (mgr.14162) 525 : cluster [DBG] pgmap v493: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:08.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:05.672414+0000 mgr.smithi078.zidbgq (mgr.14162) 525 : cluster [DBG] pgmap v493: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:09.276 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:09.513 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:09.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:07.672953+0000 mgr.smithi078.zidbgq (mgr.14162) 526 : cluster [DBG] pgmap v494: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:10.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:07.672953+0000 mgr.smithi078.zidbgq (mgr.14162) 526 : cluster [DBG] pgmap v494: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.543+0000 7f70efc81700 1 -- 172.21.15.78:0/215764596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70e80f6960 msgr2=0x7f70e80f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.543+0000 7f70efc81700 1 --2- 172.21.15.78:0/215764596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70e80f6960 0x7f70e80f6dc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f70d8004660 tx=0x7f70d8009b30).stop 2023-12-21T03:55:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.543+0000 7f70efc81700 1 -- 172.21.15.78:0/215764596 shutdown_connections 2023-12-21T03:55:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.543+0000 7f70efc81700 1 --2- 172.21.15.78:0/215764596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70e80f6960 0x7f70e80f6dc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.543+0000 7f70efc81700 1 --2- 172.21.15.78:0/215764596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70e80f56c0 0x7f70e80f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.543+0000 7f70efc81700 1 -- 172.21.15.78:0/215764596 >> 172.21.15.78:0/215764596 conn(0x7f70e80f0e20 msgr2=0x7f70e80f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.543+0000 7f70efc81700 1 -- 172.21.15.78:0/215764596 shutdown_connections 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.543+0000 7f70efc81700 1 -- 172.21.15.78:0/215764596 wait complete. 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70efc81700 1 Processor -- start 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70efc81700 1 -- start start 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70efc81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70e80f56c0 0x7f70e81070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70efc81700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70e80f6960 0x7f70e81075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70efc81700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e8107c10 con 0x7f70e80f56c0 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70efc81700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e81084c0 con 0x7f70e80f6960 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70eda1d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70e80f56c0 0x7f70e81070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70eda1d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70e80f56c0 0x7f70e81070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40644/0 (socket says 172.21.15.78:40644) 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70eda1d700 1 -- 172.21.15.78:0/3222055846 learned_addr learned my addr 172.21.15.78:0/3222055846 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70ed21c700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70e80f6960 0x7f70e81075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70eda1d700 1 -- 172.21.15.78:0/3222055846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70e80f6960 msgr2=0x7f70e81075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:10.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70eda1d700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70e80f6960 0x7f70e81075d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:10.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70eda1d700 1 -- 172.21.15.78:0/3222055846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d8005040 con 0x7f70e80f56c0 2023-12-21T03:55:10.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70ed21c700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70e80f6960 0x7f70e81075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:55:10.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.544+0000 7f70eda1d700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70e80f56c0 0x7f70e81070e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f70e40093f0 tx=0x7f70e4006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:10.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.545+0000 7f70de7fc700 1 -- 172.21.15.78:0/3222055846 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70e4009ea0 con 0x7f70e80f56c0 2023-12-21T03:55:10.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.545+0000 7f70efc81700 1 -- 172.21.15.78:0/3222055846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e81087a0 con 0x7f70e80f56c0 2023-12-21T03:55:10.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.545+0000 7f70de7fc700 1 -- 172.21.15.78:0/3222055846 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70e4013af0 con 0x7f70e80f56c0 2023-12-21T03:55:10.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.545+0000 7f70de7fc700 1 -- 172.21.15.78:0/3222055846 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70e400dad0 con 0x7f70e80f56c0 2023-12-21T03:55:10.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.545+0000 7f70efc81700 1 -- 172.21.15.78:0/3222055846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e8108d60 con 0x7f70e80f56c0 2023-12-21T03:55:10.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.546+0000 7f70efc81700 1 -- 172.21.15.78:0/3222055846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e8059100 con 0x7f70e80f56c0 2023-12-21T03:55:10.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.546+0000 7f70de7fc700 1 -- 172.21.15.78:0/3222055846 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f70e401a060 con 0x7f70e80f56c0 2023-12-21T03:55:10.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.546+0000 7f70de7fc700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70d405c2a0 0x7f70d405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:10.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.546+0000 7f70de7fc700 1 -- 172.21.15.78:0/3222055846 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f70e4075ba0 con 0x7f70e80f56c0 2023-12-21T03:55:10.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.546+0000 7f70ed21c700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70d405c2a0 0x7f70d405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:10.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.547+0000 7f70ed21c700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70d405c2a0 0x7f70d405e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f70d8009d60 tx=0x7f70d8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.550+0000 7f70de7fc700 1 -- 172.21.15.78:0/3222055846 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f70e4048b40 con 0x7f70e80f56c0 2023-12-21T03:55:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.760+0000 7f70efc81700 1 -- 172.21.15.78:0/3222055846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f70e8045ad0 con 0x7f70e80f56c0 2023-12-21T03:55:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.761+0000 7f70de7fc700 1 -- 172.21.15.78:0/3222055846 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f70e404c160 con 0x7f70e80f56c0 2023-12-21T03:55:10.762 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.762+0000 7f70efc81700 1 -- 172.21.15.78:0/3222055846 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70d405c2a0 msgr2=0x7f70d405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.762+0000 7f70efc81700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70d405c2a0 0x7f70d405e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f70d8009d60 tx=0x7f70d8009b30).stop 2023-12-21T03:55:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.762+0000 7f70efc81700 1 -- 172.21.15.78:0/3222055846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70e80f56c0 msgr2=0x7f70e81070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.762+0000 7f70efc81700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70e80f56c0 0x7f70e81070e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f70e40093f0 tx=0x7f70e4006ec0).stop 2023-12-21T03:55:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.762+0000 7f70efc81700 1 -- 172.21.15.78:0/3222055846 shutdown_connections 2023-12-21T03:55:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.762+0000 7f70efc81700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f70d405c2a0 0x7f70d405e760 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.762+0000 7f70efc81700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70e80f56c0 0x7f70e81070e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.762+0000 7f70efc81700 1 --2- 172.21.15.78:0/3222055846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70e80f6960 0x7f70e81075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.762+0000 7f70efc81700 1 -- 172.21.15.78:0/3222055846 >> 172.21.15.78:0/3222055846 conn(0x7f70e80f0e20 msgr2=0x7f70e8060380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.763+0000 7f70efc81700 1 -- 172.21.15.78:0/3222055846 shutdown_connections 2023-12-21T03:55:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:10.763+0000 7f70efc81700 1 -- 172.21.15.78:0/3222055846 wait complete. 2023-12-21T03:55:11.169 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:09.673257+0000 mgr.smithi078.zidbgq (mgr.14162) 527 : cluster [DBG] pgmap v495: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:10.762154+0000 mon.smithi078 (mon.0) 546 : audit [DBG] from='client.? 172.21.15.78:0/3222055846' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:12.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:09.673257+0000 mgr.smithi078.zidbgq (mgr.14162) 527 : cluster [DBG] pgmap v495: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:12.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:10.762154+0000 mon.smithi078 (mon.0) 546 : audit [DBG] from='client.? 172.21.15.78:0/3222055846' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:11.673613+0000 mgr.smithi078.zidbgq (mgr.14162) 528 : cluster [DBG] pgmap v496: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:14.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:11.673613+0000 mgr.smithi078.zidbgq (mgr.14162) 528 : cluster [DBG] pgmap v496: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:14.170 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:14.411 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:15.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.373+0000 7f57ccf80700 1 -- 172.21.15.78:0/151219687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c80f67c0 msgr2=0x7f57c80f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:15.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.373+0000 7f57ccf80700 1 --2- 172.21.15.78:0/151219687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c80f67c0 0x7f57c80f6be0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f57bc004660 tx=0x7f57bc009b30).stop 2023-12-21T03:55:15.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.373+0000 7f57ccf80700 1 -- 172.21.15.78:0/151219687 shutdown_connections 2023-12-21T03:55:15.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.373+0000 7f57ccf80700 1 --2- 172.21.15.78:0/151219687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57c80f7a60 0x7f57c80f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:15.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.373+0000 7f57ccf80700 1 --2- 172.21.15.78:0/151219687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c80f67c0 0x7f57c80f6be0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:15.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.373+0000 7f57ccf80700 1 -- 172.21.15.78:0/151219687 >> 172.21.15.78:0/151219687 conn(0x7f57c80f1f20 msgr2=0x7f57c80f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:15.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.373+0000 7f57ccf80700 1 -- 172.21.15.78:0/151219687 shutdown_connections 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.373+0000 7f57ccf80700 1 -- 172.21.15.78:0/151219687 wait complete. 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.373+0000 7f57ccf80700 1 Processor -- start 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.373+0000 7f57ccf80700 1 -- start start 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57ccf80700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c80f67c0 0x7f57c81070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57ccf80700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57c80f7a60 0x7f57c8107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57ccf80700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c8107b40 con 0x7f57c80f67c0 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57ccf80700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c8107c80 con 0x7f57c80f7a60 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57c659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c80f67c0 0x7f57c81070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57c659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c80f67c0 0x7f57c81070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40662/0 (socket says 172.21.15.78:40662) 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57c659c700 1 -- 172.21.15.78:0/3258632885 learned_addr learned my addr 172.21.15.78:0/3258632885 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57c5d9b700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57c80f7a60 0x7f57c8107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57c659c700 1 -- 172.21.15.78:0/3258632885 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57c80f7a60 msgr2=0x7f57c8107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57c659c700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57c80f7a60 0x7f57c8107590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57c659c700 1 -- 172.21.15.78:0/3258632885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc005040 con 0x7f57c80f67c0 2023-12-21T03:55:15.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57c5d9b700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57c80f7a60 0x7f57c8107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:55:15.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57c659c700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c80f67c0 0x7f57c81070a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f57bc009f50 tx=0x7f57bc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:15.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57b6ffd700 1 -- 172.21.15.78:0/3258632885 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57bc018070 con 0x7f57c80f67c0 2023-12-21T03:55:15.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57b6ffd700 1 -- 172.21.15.78:0/3258632885 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57bc013410 con 0x7f57c80f67c0 2023-12-21T03:55:15.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57ccf80700 1 -- 172.21.15.78:0/3258632885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c81085c0 con 0x7f57c80f67c0 2023-12-21T03:55:15.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57b6ffd700 1 -- 172.21.15.78:0/3258632885 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57bc01c620 con 0x7f57c80f67c0 2023-12-21T03:55:15.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.374+0000 7f57ccf80700 1 -- 172.21.15.78:0/3258632885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c8108b20 con 0x7f57c80f67c0 2023-12-21T03:55:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.375+0000 7f57ccf80700 1 -- 172.21.15.78:0/3258632885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c8045ad0 con 0x7f57c80f67c0 2023-12-21T03:55:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.376+0000 7f57b6ffd700 1 -- 172.21.15.78:0/3258632885 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f57bc01d420 con 0x7f57c80f67c0 2023-12-21T03:55:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.376+0000 7f57b6ffd700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57b005c2a0 0x7f57b005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.376+0000 7f57b6ffd700 1 -- 172.21.15.78:0/3258632885 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f57bc0779f0 con 0x7f57c80f67c0 2023-12-21T03:55:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.376+0000 7f57c5d9b700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57b005c2a0 0x7f57b005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.376+0000 7f57c5d9b700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57b005c2a0 0x7f57b005e760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f57b8004a70 tx=0x7f57b8009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:15.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.380+0000 7f57b6ffd700 1 -- 172.21.15.78:0/3258632885 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57bc079e60 con 0x7f57c80f67c0 2023-12-21T03:55:15.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.591+0000 7f57ccf80700 1 -- 172.21.15.78:0/3258632885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f57c8027a00 con 0x7f57c80f67c0 2023-12-21T03:55:15.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.591+0000 7f57b6ffd700 1 -- 172.21.15.78:0/3258632885 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f57bc04a9f0 con 0x7f57c80f67c0 2023-12-21T03:55:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:15.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.593+0000 7f57ccf80700 1 -- 172.21.15.78:0/3258632885 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57b005c2a0 msgr2=0x7f57b005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:15.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.593+0000 7f57ccf80700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57b005c2a0 0x7f57b005e760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f57b8004a70 tx=0x7f57b8009040).stop 2023-12-21T03:55:15.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.593+0000 7f57ccf80700 1 -- 172.21.15.78:0/3258632885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c80f67c0 msgr2=0x7f57c81070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:15.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.593+0000 7f57ccf80700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c80f67c0 0x7f57c81070a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f57bc009f50 tx=0x7f57bc009b30).stop 2023-12-21T03:55:15.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.593+0000 7f57ccf80700 1 -- 172.21.15.78:0/3258632885 shutdown_connections 2023-12-21T03:55:15.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.593+0000 7f57ccf80700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f57b005c2a0 0x7f57b005e760 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:15.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.593+0000 7f57ccf80700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c80f67c0 0x7f57c81070a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:15.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.593+0000 7f57ccf80700 1 --2- 172.21.15.78:0/3258632885 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57c80f7a60 0x7f57c8107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:15.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.593+0000 7f57ccf80700 1 -- 172.21.15.78:0/3258632885 >> 172.21.15.78:0/3258632885 conn(0x7f57c80f1f20 msgr2=0x7f57c80fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:15.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.593+0000 7f57ccf80700 1 -- 172.21.15.78:0/3258632885 shutdown_connections 2023-12-21T03:55:15.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:15.593+0000 7f57ccf80700 1 -- 172.21.15.78:0/3258632885 wait complete. 2023-12-21T03:55:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:13.674072+0000 mgr.smithi078.zidbgq (mgr.14162) 529 : cluster [DBG] pgmap v497: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:15.958 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:16.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:13.674072+0000 mgr.smithi078.zidbgq (mgr.14162) 529 : cluster [DBG] pgmap v497: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:16.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:16 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:15.592599+0000 mon.smithi078 (mon.0) 547 : audit [DBG] from='client.? 172.21.15.78:0/3258632885' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:17.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:16 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:15.592599+0000 mon.smithi078 (mon.0) 547 : audit [DBG] from='client.? 172.21.15.78:0/3258632885' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:17.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:15.674401+0000 mgr.smithi078.zidbgq (mgr.14162) 530 : cluster [DBG] pgmap v498: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:17.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:15.674401+0000 mgr.smithi078.zidbgq (mgr.14162) 530 : cluster [DBG] pgmap v498: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:18.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:19.194 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:19.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:17.674766+0000 mgr.smithi078.zidbgq (mgr.14162) 531 : cluster [DBG] pgmap v499: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:20.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:17.674766+0000 mgr.smithi078.zidbgq (mgr.14162) 531 : cluster [DBG] pgmap v499: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:20.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.221+0000 7f4f9c245700 1 -- 172.21.15.78:0/2917622135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940f67c0 msgr2=0x7f4f940f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:20.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.221+0000 7f4f9c245700 1 --2- 172.21.15.78:0/2917622135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940f67c0 0x7f4f940f6be0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f4f88004660 tx=0x7f4f88009b30).stop 2023-12-21T03:55:20.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.221+0000 7f4f9c245700 1 -- 172.21.15.78:0/2917622135 shutdown_connections 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.221+0000 7f4f9c245700 1 --2- 172.21.15.78:0/2917622135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f940f7a60 0x7f4f940f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.221+0000 7f4f9c245700 1 --2- 172.21.15.78:0/2917622135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940f67c0 0x7f4f940f6be0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.221+0000 7f4f9c245700 1 -- 172.21.15.78:0/2917622135 >> 172.21.15.78:0/2917622135 conn(0x7f4f940f1f20 msgr2=0x7f4f940f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.221+0000 7f4f9c245700 1 -- 172.21.15.78:0/2917622135 shutdown_connections 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.221+0000 7f4f9c245700 1 -- 172.21.15.78:0/2917622135 wait complete. 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f9c245700 1 Processor -- start 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f9c245700 1 -- start start 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f9c245700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940f67c0 0x7f4f9406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f9c245700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f940f7a60 0x7f4f94069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f9c245700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9406a410 con 0x7f4f940f67c0 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f9c245700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9406a550 con 0x7f4f940f7a60 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f99fe1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940f67c0 0x7f4f9406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:20.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f99fe1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940f67c0 0x7f4f9406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58856/0 (socket says 172.21.15.78:58856) 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f99fe1700 1 -- 172.21.15.78:0/580573454 learned_addr learned my addr 172.21.15.78:0/580573454 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f997e0700 1 --2- 172.21.15.78:0/580573454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f940f7a60 0x7f4f94069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f99fe1700 1 -- 172.21.15.78:0/580573454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f940f7a60 msgr2=0x7f4f94069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f99fe1700 1 --2- 172.21.15.78:0/580573454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f940f7a60 0x7f4f94069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f99fe1700 1 -- 172.21.15.78:0/580573454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f88005040 con 0x7f4f940f67c0 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.222+0000 7f4f99fe1700 1 --2- 172.21.15.78:0/580573454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940f67c0 0x7f4f9406b9a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f4f88009f50 tx=0x7f4f88009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.223+0000 7f4f867fc700 1 -- 172.21.15.78:0/580573454 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f88018070 con 0x7f4f940f67c0 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.223+0000 7f4f9c245700 1 -- 172.21.15.78:0/580573454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9406a7d0 con 0x7f4f940f67c0 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.223+0000 7f4f867fc700 1 -- 172.21.15.78:0/580573454 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f88013410 con 0x7f4f940f67c0 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.223+0000 7f4f867fc700 1 -- 172.21.15.78:0/580573454 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8801c620 con 0x7f4f940f67c0 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.223+0000 7f4f9c245700 1 -- 172.21.15.78:0/580573454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9410f100 con 0x7f4f940f67c0 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.224+0000 7f4f9c245700 1 -- 172.21.15.78:0/580573454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f94045ad0 con 0x7f4f940f67c0 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.224+0000 7f4f867fc700 1 -- 172.21.15.78:0/580573454 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4f8801d420 con 0x7f4f940f67c0 2023-12-21T03:55:20.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.224+0000 7f4f867fc700 1 --2- 172.21.15.78:0/580573454 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4f8005c2a0 0x7f4f8005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:20.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.224+0000 7f4f867fc700 1 -- 172.21.15.78:0/580573454 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4f880779f0 con 0x7f4f940f67c0 2023-12-21T03:55:20.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.224+0000 7f4f997e0700 1 --2- 172.21.15.78:0/580573454 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4f8005c2a0 0x7f4f8005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:20.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.225+0000 7f4f997e0700 1 --2- 172.21.15.78:0/580573454 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4f8005c2a0 0x7f4f8005e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f4f9000b830 tx=0x7f4f90008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:20.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.228+0000 7f4f867fc700 1 -- 172.21.15.78:0/580573454 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f88079e60 con 0x7f4f940f67c0 2023-12-21T03:55:20.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.436+0000 7f4f9c245700 1 -- 172.21.15.78:0/580573454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4f94027a00 con 0x7f4f940f67c0 2023-12-21T03:55:20.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.437+0000 7f4f867fc700 1 -- 172.21.15.78:0/580573454 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f4f8804a9f0 con 0x7f4f940f67c0 2023-12-21T03:55:20.439 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:20.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.438+0000 7f4f9c245700 1 -- 172.21.15.78:0/580573454 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4f8005c2a0 msgr2=0x7f4f8005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:20.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.438+0000 7f4f9c245700 1 --2- 172.21.15.78:0/580573454 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4f8005c2a0 0x7f4f8005e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f4f9000b830 tx=0x7f4f90008040).stop 2023-12-21T03:55:20.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.438+0000 7f4f9c245700 1 -- 172.21.15.78:0/580573454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940f67c0 msgr2=0x7f4f9406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:20.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.438+0000 7f4f9c245700 1 --2- 172.21.15.78:0/580573454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940f67c0 0x7f4f9406b9a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f4f88009f50 tx=0x7f4f88009b30).stop 2023-12-21T03:55:20.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.438+0000 7f4f9c245700 1 -- 172.21.15.78:0/580573454 shutdown_connections 2023-12-21T03:55:20.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.438+0000 7f4f9c245700 1 --2- 172.21.15.78:0/580573454 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f4f8005c2a0 0x7f4f8005e760 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:20.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.438+0000 7f4f9c245700 1 --2- 172.21.15.78:0/580573454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940f67c0 0x7f4f9406b9a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:20.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.438+0000 7f4f9c245700 1 --2- 172.21.15.78:0/580573454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f940f7a60 0x7f4f94069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:20.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.438+0000 7f4f9c245700 1 -- 172.21.15.78:0/580573454 >> 172.21.15.78:0/580573454 conn(0x7f4f940f1f20 msgr2=0x7f4f940fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:20.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.439+0000 7f4f9c245700 1 -- 172.21.15.78:0/580573454 shutdown_connections 2023-12-21T03:55:20.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:20.439+0000 7f4f9c245700 1 -- 172.21.15.78:0/580573454 wait complete. 2023-12-21T03:55:20.715 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:20 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:20.438204+0000 mon.smithi078 (mon.0) 548 : audit [DBG] from='client.? 172.21.15.78:0/580573454' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:20.832 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:21.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:20 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:20.438204+0000 mon.smithi078 (mon.0) 548 : audit [DBG] from='client.? 172.21.15.78:0/580573454' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:21.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:19.675221+0000 mgr.smithi078.zidbgq (mgr.14162) 532 : cluster [DBG] pgmap v500: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:22.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:19.675221+0000 mgr.smithi078.zidbgq (mgr.14162) 532 : cluster [DBG] pgmap v500: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:23.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:21.675571+0000 mgr.smithi078.zidbgq (mgr.14162) 533 : cluster [DBG] pgmap v501: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:23.834 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:24.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:21.675571+0000 mgr.smithi078.zidbgq (mgr.14162) 533 : cluster [DBG] pgmap v501: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.030+0000 7f506a0a9700 1 -- 172.21.15.78:0/33919828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50640f7a70 msgr2=0x7f50640f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.030+0000 7f506a0a9700 1 --2- 172.21.15.78:0/33919828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50640f7a70 0x7f50640f7ed0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f5054004660 tx=0x7f5054009b30).stop 2023-12-21T03:55:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.030+0000 7f506a0a9700 1 -- 172.21.15.78:0/33919828 shutdown_connections 2023-12-21T03:55:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.030+0000 7f506a0a9700 1 --2- 172.21.15.78:0/33919828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50640f7a70 0x7f50640f7ed0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.030+0000 7f506a0a9700 1 --2- 172.21.15.78:0/33919828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50640f67d0 0x7f50640f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.030+0000 7f506a0a9700 1 -- 172.21.15.78:0/33919828 >> 172.21.15.78:0/33919828 conn(0x7f50640f1f50 msgr2=0x7f50640f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.030+0000 7f506a0a9700 1 -- 172.21.15.78:0/33919828 shutdown_connections 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.030+0000 7f506a0a9700 1 -- 172.21.15.78:0/33919828 wait complete. 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.030+0000 7f506a0a9700 1 Processor -- start 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f506a0a9700 1 -- start start 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f506a0a9700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50640f67d0 0x7f50641070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f506a0a9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50640f7a70 0x7f5064107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f506a0a9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5064107b40 con 0x7f50640f7a70 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f506a0a9700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5064107c80 con 0x7f50640f67d0 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f5062ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50640f7a70 0x7f5064107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f5062ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50640f7a70 0x7f5064107590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58860/0 (socket says 172.21.15.78:58860) 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f5062ffd700 1 -- 172.21.15.78:0/18654959 learned_addr learned my addr 172.21.15.78:0/18654959 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f5062ffd700 1 -- 172.21.15.78:0/18654959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50640f67d0 msgr2=0x7f50641070a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f50637fe700 1 --2- 172.21.15.78:0/18654959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50640f67d0 0x7f50641070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f5062ffd700 1 --2- 172.21.15.78:0/18654959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50640f67d0 0x7f50641070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:25.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f5062ffd700 1 -- 172.21.15.78:0/18654959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5054005040 con 0x7f50640f7a70 2023-12-21T03:55:25.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f5062ffd700 1 --2- 172.21.15.78:0/18654959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50640f7a70 0x7f5064107590 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f5054009f50 tx=0x7f5054009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:25.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f504bfff700 1 -- 172.21.15.78:0/18654959 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5054018070 con 0x7f50640f7a70 2023-12-21T03:55:25.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f506a0a9700 1 -- 172.21.15.78:0/18654959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50641085c0 con 0x7f50640f7a70 2023-12-21T03:55:25.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f504bfff700 1 -- 172.21.15.78:0/18654959 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5054013410 con 0x7f50640f7a70 2023-12-21T03:55:25.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f504bfff700 1 -- 172.21.15.78:0/18654959 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f505401c3c0 con 0x7f50640f7a70 2023-12-21T03:55:25.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.031+0000 7f506a0a9700 1 -- 172.21.15.78:0/18654959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5064108b20 con 0x7f50640f7a70 2023-12-21T03:55:25.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.033+0000 7f506a0a9700 1 -- 172.21.15.78:0/18654959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5064045ad0 con 0x7f50640f7a70 2023-12-21T03:55:25.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.034+0000 7f504bfff700 1 -- 172.21.15.78:0/18654959 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f505401d420 con 0x7f50640f7a70 2023-12-21T03:55:25.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.034+0000 7f504bfff700 1 --2- 172.21.15.78:0/18654959 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f504c05c2a0 0x7f504c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:25.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.034+0000 7f504bfff700 1 -- 172.21.15.78:0/18654959 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5054077ea0 con 0x7f50640f7a70 2023-12-21T03:55:25.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.034+0000 7f50637fe700 1 --2- 172.21.15.78:0/18654959 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f504c05c2a0 0x7f504c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:25.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.034+0000 7f50637fe700 1 --2- 172.21.15.78:0/18654959 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f504c05c2a0 0x7f504c05e760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f50640f78d0 tx=0x7f505800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:25.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.037+0000 7f504bfff700 1 -- 172.21.15.78:0/18654959 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f505404af90 con 0x7f50640f7a70 2023-12-21T03:55:25.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.247+0000 7f506a0a9700 1 -- 172.21.15.78:0/18654959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f50640ff060 con 0x7f50640f7a70 2023-12-21T03:55:25.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.248+0000 7f504bfff700 1 -- 172.21.15.78:0/18654959 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f505404e5b0 con 0x7f50640f7a70 2023-12-21T03:55:25.249 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:25.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.249+0000 7f506a0a9700 1 -- 172.21.15.78:0/18654959 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f504c05c2a0 msgr2=0x7f504c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:25.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.249+0000 7f506a0a9700 1 --2- 172.21.15.78:0/18654959 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f504c05c2a0 0x7f504c05e760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f50640f78d0 tx=0x7f505800b040).stop 2023-12-21T03:55:25.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.249+0000 7f506a0a9700 1 -- 172.21.15.78:0/18654959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50640f7a70 msgr2=0x7f5064107590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:25.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.249+0000 7f506a0a9700 1 --2- 172.21.15.78:0/18654959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50640f7a70 0x7f5064107590 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f5054009f50 tx=0x7f5054009b30).stop 2023-12-21T03:55:25.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.249+0000 7f506a0a9700 1 -- 172.21.15.78:0/18654959 shutdown_connections 2023-12-21T03:55:25.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.249+0000 7f506a0a9700 1 --2- 172.21.15.78:0/18654959 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f504c05c2a0 0x7f504c05e760 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:25.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.249+0000 7f506a0a9700 1 --2- 172.21.15.78:0/18654959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50640f67d0 0x7f50641070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:25.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.249+0000 7f506a0a9700 1 --2- 172.21.15.78:0/18654959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50640f7a70 0x7f5064107590 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:25.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.249+0000 7f506a0a9700 1 -- 172.21.15.78:0/18654959 >> 172.21.15.78:0/18654959 conn(0x7f50640f1f50 msgr2=0x7f50640fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:25.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.249+0000 7f506a0a9700 1 -- 172.21.15.78:0/18654959 shutdown_connections 2023-12-21T03:55:25.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:25.249+0000 7f506a0a9700 1 -- 172.21.15.78:0/18654959 wait complete. 2023-12-21T03:55:25.614 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:23.676007+0000 mgr.smithi078.zidbgq (mgr.14162) 534 : cluster [DBG] pgmap v502: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:25.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:25.249008+0000 mon.smithi078 (mon.0) 549 : audit [DBG] from='client.? 172.21.15.78:0/18654959' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:26.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:23.676007+0000 mgr.smithi078.zidbgq (mgr.14162) 534 : cluster [DBG] pgmap v502: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:26.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:25.249008+0000 mon.smithi078 (mon.0) 549 : audit [DBG] from='client.? 172.21.15.78:0/18654959' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:25.676464+0000 mgr.smithi078.zidbgq (mgr.14162) 535 : cluster [DBG] pgmap v503: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:28.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:25.676464+0000 mgr.smithi078.zidbgq (mgr.14162) 535 : cluster [DBG] pgmap v503: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:28.615 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:29.627 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:27.676925+0000 mgr.smithi078.zidbgq (mgr.14162) 536 : cluster [DBG] pgmap v504: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:29.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.812+0000 7f03d7058700 1 -- 172.21.15.78:0/1502058224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d00f56e0 msgr2=0x7f03d00f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:29.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.812+0000 7f03d7058700 1 --2- 172.21.15.78:0/1502058224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d00f56e0 0x7f03d00f5b00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f03c4004660 tx=0x7f03c4009b30).stop 2023-12-21T03:55:29.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.812+0000 7f03d7058700 1 -- 172.21.15.78:0/1502058224 shutdown_connections 2023-12-21T03:55:29.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.812+0000 7f03d7058700 1 --2- 172.21.15.78:0/1502058224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d00f6980 0x7f03d00f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:29.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.812+0000 7f03d7058700 1 --2- 172.21.15.78:0/1502058224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d00f56e0 0x7f03d00f5b00 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:29.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.812+0000 7f03d7058700 1 -- 172.21.15.78:0/1502058224 >> 172.21.15.78:0/1502058224 conn(0x7f03d00f0e20 msgr2=0x7f03d00f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:29.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.812+0000 7f03d7058700 1 -- 172.21.15.78:0/1502058224 shutdown_connections 2023-12-21T03:55:29.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.812+0000 7f03d7058700 1 -- 172.21.15.78:0/1502058224 wait complete. 2023-12-21T03:55:29.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.813+0000 7f03d7058700 1 Processor -- start 2023-12-21T03:55:29.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.813+0000 7f03d7058700 1 -- start start 2023-12-21T03:55:29.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.813+0000 7f03d7058700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d00f56e0 0x7f03d01070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:29.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.813+0000 7f03d7058700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d00f6980 0x7f03d01075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:29.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.813+0000 7f03d7058700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d0107bf0 con 0x7f03d00f56e0 2023-12-21T03:55:29.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.813+0000 7f03d7058700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d01084a0 con 0x7f03d00f6980 2023-12-21T03:55:29.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.813+0000 7f03d4df4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d00f56e0 0x7f03d01070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:29.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.813+0000 7f03d4df4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d00f56e0 0x7f03d01070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60382/0 (socket says 172.21.15.78:60382) 2023-12-21T03:55:29.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.813+0000 7f03d4df4700 1 -- 172.21.15.78:0/1648381526 learned_addr learned my addr 172.21.15.78:0/1648381526 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:29.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.814+0000 7f03d4df4700 1 -- 172.21.15.78:0/1648381526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d00f6980 msgr2=0x7f03d01075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.814+0000 7f03cffff700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d00f6980 0x7f03d01075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.814+0000 7f03d4df4700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d00f6980 0x7f03d01075b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.814+0000 7f03d4df4700 1 -- 172.21.15.78:0/1648381526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c4005040 con 0x7f03d00f56e0 2023-12-21T03:55:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.814+0000 7f03cffff700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d00f6980 0x7f03d01075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:55:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.814+0000 7f03d4df4700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d00f56e0 0x7f03d01070c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f03c4009f50 tx=0x7f03c4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.814+0000 7f03cd7fa700 1 -- 172.21.15.78:0/1648381526 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03c4018070 con 0x7f03d00f56e0 2023-12-21T03:55:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.814+0000 7f03cd7fa700 1 -- 172.21.15.78:0/1648381526 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03c4013410 con 0x7f03d00f56e0 2023-12-21T03:55:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.814+0000 7f03d7058700 1 -- 172.21.15.78:0/1648381526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d0108720 con 0x7f03d00f56e0 2023-12-21T03:55:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.814+0000 7f03cd7fa700 1 -- 172.21.15.78:0/1648381526 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03c401c620 con 0x7f03d00f56e0 2023-12-21T03:55:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.814+0000 7f03d7058700 1 -- 172.21.15.78:0/1648381526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d0108c80 con 0x7f03d00f56e0 2023-12-21T03:55:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.815+0000 7f03d7058700 1 -- 172.21.15.78:0/1648381526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d0059100 con 0x7f03d00f56e0 2023-12-21T03:55:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.815+0000 7f03cd7fa700 1 -- 172.21.15.78:0/1648381526 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f03c401d420 con 0x7f03d00f56e0 2023-12-21T03:55:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.816+0000 7f03cd7fa700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03c005c2a0 0x7f03c005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:29.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.816+0000 7f03cd7fa700 1 -- 172.21.15.78:0/1648381526 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f03c4077a40 con 0x7f03d00f56e0 2023-12-21T03:55:29.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.816+0000 7f03cffff700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03c005c2a0 0x7f03c005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:29.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.816+0000 7f03cffff700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03c005c2a0 0x7f03c005e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f03bc00a7e0 tx=0x7f03bc008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:29.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:29.820+0000 7f03cd7fa700 1 -- 172.21.15.78:0/1648381526 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03c404aa40 con 0x7f03d00f56e0 2023-12-21T03:55:30.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:27.676925+0000 mgr.smithi078.zidbgq (mgr.14162) 536 : cluster [DBG] pgmap v504: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:30.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.027+0000 7f03d7058700 1 -- 172.21.15.78:0/1648381526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f03d01092c0 con 0x7f03d00f56e0 2023-12-21T03:55:30.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.028+0000 7f03cd7fa700 1 -- 172.21.15.78:0/1648381526 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f03c404e060 con 0x7f03d00f56e0 2023-12-21T03:55:30.028 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:30.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.029+0000 7f03d7058700 1 -- 172.21.15.78:0/1648381526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03c005c2a0 msgr2=0x7f03c005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:30.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.029+0000 7f03d7058700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03c005c2a0 0x7f03c005e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f03bc00a7e0 tx=0x7f03bc008040).stop 2023-12-21T03:55:30.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.029+0000 7f03d7058700 1 -- 172.21.15.78:0/1648381526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d00f56e0 msgr2=0x7f03d01070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:30.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.029+0000 7f03d7058700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d00f56e0 0x7f03d01070c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f03c4009f50 tx=0x7f03c4009b30).stop 2023-12-21T03:55:30.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.029+0000 7f03d7058700 1 -- 172.21.15.78:0/1648381526 shutdown_connections 2023-12-21T03:55:30.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.029+0000 7f03d7058700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f03c005c2a0 0x7f03c005e760 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:30.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.029+0000 7f03d7058700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d00f56e0 0x7f03d01070c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:30.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.029+0000 7f03d7058700 1 --2- 172.21.15.78:0/1648381526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d00f6980 0x7f03d01075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:30.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.029+0000 7f03d7058700 1 -- 172.21.15.78:0/1648381526 >> 172.21.15.78:0/1648381526 conn(0x7f03d00f0e20 msgr2=0x7f03d00603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:30.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.029+0000 7f03d7058700 1 -- 172.21.15.78:0/1648381526 shutdown_connections 2023-12-21T03:55:30.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:30.029+0000 7f03d7058700 1 -- 172.21.15.78:0/1648381526 wait complete. 2023-12-21T03:55:30.414 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:30.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:30.028141+0000 mon.smithi078 (mon.0) 550 : audit [DBG] from='client.? 172.21.15.78:0/1648381526' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:31.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:30.028141+0000 mon.smithi078 (mon.0) 550 : audit [DBG] from='client.? 172.21.15.78:0/1648381526' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:32.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:29.677313+0000 mgr.smithi078.zidbgq (mgr.14162) 537 : cluster [DBG] pgmap v505: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:32.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:30.571026+0000 mon.smithi078 (mon.0) 551 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:55:32.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:30.571328+0000 mon.smithi078 (mon.0) 552 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:55:32.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:29.677313+0000 mgr.smithi078.zidbgq (mgr.14162) 537 : cluster [DBG] pgmap v505: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:32.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:30.571026+0000 mon.smithi078 (mon.0) 551 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:55:32.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:30.571328+0000 mon.smithi078 (mon.0) 552 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:55:33.415 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:33.650 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:33.792 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:31.677648+0000 mgr.smithi078.zidbgq (mgr.14162) 538 : cluster [DBG] pgmap v506: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:34.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:31.677648+0000 mgr.smithi078.zidbgq (mgr.14162) 538 : cluster [DBG] pgmap v506: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:34.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.634+0000 7fa7aa051700 1 -- 172.21.15.78:0/249413122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7a40f6980 msgr2=0x7fa7a40f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.634+0000 7fa7aa051700 1 --2- 172.21.15.78:0/249413122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7a40f6980 0x7fa7a40f6de0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fa78c004660 tx=0x7fa78c009b30).stop 2023-12-21T03:55:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.634+0000 7fa7aa051700 1 -- 172.21.15.78:0/249413122 shutdown_connections 2023-12-21T03:55:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.634+0000 7fa7aa051700 1 --2- 172.21.15.78:0/249413122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7a40f6980 0x7fa7a40f6de0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.634+0000 7fa7aa051700 1 --2- 172.21.15.78:0/249413122 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7a40f56e0 0x7fa7a40f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.634+0000 7fa7aa051700 1 -- 172.21.15.78:0/249413122 >> 172.21.15.78:0/249413122 conn(0x7fa7a40f0e20 msgr2=0x7fa7a40f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.634+0000 7fa7aa051700 1 -- 172.21.15.78:0/249413122 shutdown_connections 2023-12-21T03:55:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.634+0000 7fa7aa051700 1 -- 172.21.15.78:0/249413122 wait complete. 2023-12-21T03:55:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.635+0000 7fa7aa051700 1 Processor -- start 2023-12-21T03:55:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.635+0000 7fa7aa051700 1 -- start start 2023-12-21T03:55:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.635+0000 7fa7aa051700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7a40f56e0 0x7fa7a4109f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.635+0000 7fa7aa051700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7a4060810 0x7fa7a410c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.635+0000 7fa7aa051700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a410ca40 con 0x7fa7a4060810 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.635+0000 7fa7aa051700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a410cb80 con 0x7fa7a40f56e0 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.635+0000 7fa7a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7a4060810 0x7fa7a410c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.635+0000 7fa7a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7a4060810 0x7fa7a410c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60408/0 (socket says 172.21.15.78:60408) 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.635+0000 7fa7a2ffd700 1 -- 172.21.15.78:0/1687236899 learned_addr learned my addr 172.21.15.78:0/1687236899 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.635+0000 7fa7a37fe700 1 --2- 172.21.15.78:0/1687236899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7a40f56e0 0x7fa7a4109f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.636+0000 7fa7a2ffd700 1 -- 172.21.15.78:0/1687236899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7a40f56e0 msgr2=0x7fa7a4109f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.636+0000 7fa7a2ffd700 1 --2- 172.21.15.78:0/1687236899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7a40f56e0 0x7fa7a4109f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.636+0000 7fa7a2ffd700 1 -- 172.21.15.78:0/1687236899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c005040 con 0x7fa7a4060810 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.636+0000 7fa7a2ffd700 1 --2- 172.21.15.78:0/1687236899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7a4060810 0x7fa7a410c470 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fa78c009f50 tx=0x7fa78c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.636+0000 7fa79bfff700 1 -- 172.21.15.78:0/1687236899 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa78c019070 con 0x7fa7a4060810 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.636+0000 7fa7aa051700 1 -- 172.21.15.78:0/1687236899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a410ce00 con 0x7fa7a4060810 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.636+0000 7fa79bfff700 1 -- 172.21.15.78:0/1687236899 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa78c013410 con 0x7fa7a4060810 2023-12-21T03:55:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.636+0000 7fa79bfff700 1 -- 172.21.15.78:0/1687236899 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa78c014650 con 0x7fa7a4060810 2023-12-21T03:55:34.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.636+0000 7fa7aa051700 1 -- 172.21.15.78:0/1687236899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a410d2d0 con 0x7fa7a4060810 2023-12-21T03:55:34.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.637+0000 7fa7aa051700 1 -- 172.21.15.78:0/1687236899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a4045ad0 con 0x7fa7a4060810 2023-12-21T03:55:34.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.637+0000 7fa79bfff700 1 -- 172.21.15.78:0/1687236899 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa78c0147b0 con 0x7fa7a4060810 2023-12-21T03:55:34.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.638+0000 7fa79bfff700 1 --2- 172.21.15.78:0/1687236899 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa79005c2a0 0x7fa79005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:34.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.638+0000 7fa79bfff700 1 -- 172.21.15.78:0/1687236899 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa78c077080 con 0x7fa7a4060810 2023-12-21T03:55:34.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.638+0000 7fa7a37fe700 1 --2- 172.21.15.78:0/1687236899 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa79005c2a0 0x7fa79005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:34.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.638+0000 7fa7a37fe700 1 --2- 172.21.15.78:0/1687236899 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa79005c2a0 0x7fa79005e760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fa7a40605a0 tx=0x7fa794008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.642+0000 7fa79bfff700 1 -- 172.21.15.78:0/1687236899 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa78c04a0d0 con 0x7fa7a4060810 2023-12-21T03:55:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.849+0000 7fa7aa051700 1 -- 172.21.15.78:0/1687236899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa7a4027a00 con 0x7fa7a4060810 2023-12-21T03:55:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.850+0000 7fa79bfff700 1 -- 172.21.15.78:0/1687236899 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fa78c04d6f0 con 0x7fa7a4060810 2023-12-21T03:55:34.850 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.851+0000 7fa7aa051700 1 -- 172.21.15.78:0/1687236899 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa79005c2a0 msgr2=0x7fa79005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.851+0000 7fa7aa051700 1 --2- 172.21.15.78:0/1687236899 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa79005c2a0 0x7fa79005e760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fa7a40605a0 tx=0x7fa794008040).stop 2023-12-21T03:55:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.852+0000 7fa7aa051700 1 -- 172.21.15.78:0/1687236899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7a4060810 msgr2=0x7fa7a410c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.852+0000 7fa7aa051700 1 --2- 172.21.15.78:0/1687236899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7a4060810 0x7fa7a410c470 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fa78c009f50 tx=0x7fa78c009b30).stop 2023-12-21T03:55:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.852+0000 7fa7aa051700 1 -- 172.21.15.78:0/1687236899 shutdown_connections 2023-12-21T03:55:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.852+0000 7fa7aa051700 1 --2- 172.21.15.78:0/1687236899 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa79005c2a0 0x7fa79005e760 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.852+0000 7fa7aa051700 1 --2- 172.21.15.78:0/1687236899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7a40f56e0 0x7fa7a4109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.852+0000 7fa7aa051700 1 --2- 172.21.15.78:0/1687236899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7a4060810 0x7fa7a410c470 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.852+0000 7fa7aa051700 1 -- 172.21.15.78:0/1687236899 >> 172.21.15.78:0/1687236899 conn(0x7fa7a40f0e20 msgr2=0x7fa7a40fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.852+0000 7fa7aa051700 1 -- 172.21.15.78:0/1687236899 shutdown_connections 2023-12-21T03:55:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:34.852+0000 7fa7aa051700 1 -- 172.21.15.78:0/1687236899 wait complete. 2023-12-21T03:55:35.240 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:36.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:33.677968+0000 mgr.smithi078.zidbgq (mgr.14162) 539 : cluster [DBG] pgmap v507: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:36.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:34.850326+0000 mon.smithi078 (mon.0) 553 : audit [DBG] from='client.? 172.21.15.78:0/1687236899' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:36.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:33.677968+0000 mgr.smithi078.zidbgq (mgr.14162) 539 : cluster [DBG] pgmap v507: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:36.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:34.850326+0000 mon.smithi078 (mon.0) 553 : audit [DBG] from='client.? 172.21.15.78:0/1687236899' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:38.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:35.678297+0000 mgr.smithi078.zidbgq (mgr.14162) 540 : cluster [DBG] pgmap v508: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:38.068 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:35.678297+0000 mgr.smithi078.zidbgq (mgr.14162) 540 : cluster [DBG] pgmap v508: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:38.241 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:38.471 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.432+0000 7f8f716c7700 1 -- 172.21.15.78:0/1548598274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6c0f56e0 msgr2=0x7f8f6c0f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.432+0000 7f8f716c7700 1 --2- 172.21.15.78:0/1548598274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6c0f56e0 0x7f8f6c0f5b00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f8f60004660 tx=0x7f8f60009b30).stop 2023-12-21T03:55:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.432+0000 7f8f716c7700 1 -- 172.21.15.78:0/1548598274 shutdown_connections 2023-12-21T03:55:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.432+0000 7f8f716c7700 1 --2- 172.21.15.78:0/1548598274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0f6980 0x7f8f6c0f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.432+0000 7f8f716c7700 1 --2- 172.21.15.78:0/1548598274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6c0f56e0 0x7f8f6c0f5b00 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.432+0000 7f8f716c7700 1 -- 172.21.15.78:0/1548598274 >> 172.21.15.78:0/1548598274 conn(0x7f8f6c0f0e20 msgr2=0x7f8f6c0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.432+0000 7f8f716c7700 1 -- 172.21.15.78:0/1548598274 shutdown_connections 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.432+0000 7f8f716c7700 1 -- 172.21.15.78:0/1548598274 wait complete. 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f716c7700 1 Processor -- start 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f716c7700 1 -- start start 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f716c7700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0f56e0 0x7f8f6c109e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f716c7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6c0f6980 0x7f8f6c10c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f716c7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0609d0 con 0x7f8f6c0f6980 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f716c7700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c060b10 con 0x7f8f6c0f56e0 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f6a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6c0f6980 0x7f8f6c10c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f6a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6c0f6980 0x7f8f6c10c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44972/0 (socket says 172.21.15.78:44972) 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f6a7fc700 1 -- 172.21.15.78:0/2572616344 learned_addr learned my addr 172.21.15.78:0/2572616344 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f6a7fc700 1 -- 172.21.15.78:0/2572616344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0f56e0 msgr2=0x7f8f6c109e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f6a7fc700 1 --2- 172.21.15.78:0/2572616344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0f56e0 0x7f8f6c109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f6a7fc700 1 -- 172.21.15.78:0/2572616344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f60005040 con 0x7f8f6c0f6980 2023-12-21T03:55:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f6a7fc700 1 --2- 172.21.15.78:0/2572616344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6c0f6980 0x7f8f6c10c300 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f8f5c00b410 tx=0x7f8f5c006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f537fe700 1 -- 172.21.15.78:0/2572616344 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f5c0125f0 con 0x7f8f6c0f6980 2023-12-21T03:55:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f537fe700 1 -- 172.21.15.78:0/2572616344 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f5c012c00 con 0x7f8f6c0f6980 2023-12-21T03:55:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f537fe700 1 -- 172.21.15.78:0/2572616344 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f5c002aa0 con 0x7f8f6c0f6980 2023-12-21T03:55:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.433+0000 7f8f716c7700 1 -- 172.21.15.78:0/2572616344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c10c960 con 0x7f8f6c0f6980 2023-12-21T03:55:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.434+0000 7f8f716c7700 1 -- 172.21.15.78:0/2572616344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c10cea0 con 0x7f8f6c0f6980 2023-12-21T03:55:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.434+0000 7f8f716c7700 1 -- 172.21.15.78:0/2572616344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c059100 con 0x7f8f6c0f6980 2023-12-21T03:55:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.435+0000 7f8f537fe700 1 -- 172.21.15.78:0/2572616344 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8f5c00f030 con 0x7f8f6c0f6980 2023-12-21T03:55:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.435+0000 7f8f537fe700 1 --2- 172.21.15.78:0/2572616344 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8f5405c2a0 0x7f8f5405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.435+0000 7f8f537fe700 1 -- 172.21.15.78:0/2572616344 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8f5c076060 con 0x7f8f6c0f6980 2023-12-21T03:55:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.435+0000 7f8f6affd700 1 --2- 172.21.15.78:0/2572616344 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8f5405c2a0 0x7f8f5405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.435+0000 7f8f6affd700 1 --2- 172.21.15.78:0/2572616344 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8f5405c2a0 0x7f8f5405e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f8f60009f50 tx=0x7f8f60009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:39.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.439+0000 7f8f537fe700 1 -- 172.21.15.78:0/2572616344 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f5c049000 con 0x7f8f6c0f6980 2023-12-21T03:55:39.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.645+0000 7f8f716c7700 1 -- 172.21.15.78:0/2572616344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8f6c045ad0 con 0x7f8f6c0f6980 2023-12-21T03:55:39.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.645+0000 7f8f537fe700 1 -- 172.21.15.78:0/2572616344 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f8f5c04c620 con 0x7f8f6c0f6980 2023-12-21T03:55:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:39.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.647+0000 7f8f716c7700 1 -- 172.21.15.78:0/2572616344 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8f5405c2a0 msgr2=0x7f8f5405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:39.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.647+0000 7f8f716c7700 1 --2- 172.21.15.78:0/2572616344 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8f5405c2a0 0x7f8f5405e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f8f60009f50 tx=0x7f8f60009b30).stop 2023-12-21T03:55:39.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.647+0000 7f8f716c7700 1 -- 172.21.15.78:0/2572616344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6c0f6980 msgr2=0x7f8f6c10c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:39.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.647+0000 7f8f716c7700 1 --2- 172.21.15.78:0/2572616344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6c0f6980 0x7f8f6c10c300 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f8f5c00b410 tx=0x7f8f5c006ef0).stop 2023-12-21T03:55:39.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.647+0000 7f8f716c7700 1 -- 172.21.15.78:0/2572616344 shutdown_connections 2023-12-21T03:55:39.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.647+0000 7f8f716c7700 1 --2- 172.21.15.78:0/2572616344 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8f5405c2a0 0x7f8f5405e760 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:39.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.647+0000 7f8f716c7700 1 --2- 172.21.15.78:0/2572616344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0f56e0 0x7f8f6c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:39.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.647+0000 7f8f716c7700 1 --2- 172.21.15.78:0/2572616344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6c0f6980 0x7f8f6c10c300 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:39.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.647+0000 7f8f716c7700 1 -- 172.21.15.78:0/2572616344 >> 172.21.15.78:0/2572616344 conn(0x7f8f6c0f0e20 msgr2=0x7f8f6c0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:39.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.647+0000 7f8f716c7700 1 -- 172.21.15.78:0/2572616344 shutdown_connections 2023-12-21T03:55:39.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:39.647+0000 7f8f716c7700 1 -- 172.21.15.78:0/2572616344 wait complete. 2023-12-21T03:55:39.962 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:37.678685+0000 mgr.smithi078.zidbgq (mgr.14162) 541 : cluster [DBG] pgmap v509: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:40.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:37.678685+0000 mgr.smithi078.zidbgq (mgr.14162) 541 : cluster [DBG] pgmap v509: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:40.031 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:41.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:40 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:39.646701+0000 mon.smithi078 (mon.0) 554 : audit [DBG] from='client.? 172.21.15.78:0/2572616344' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:41.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:40 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:39.646701+0000 mon.smithi078 (mon.0) 554 : audit [DBG] from='client.? 172.21.15.78:0/2572616344' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:42.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:39.679199+0000 mgr.smithi078.zidbgq (mgr.14162) 542 : cluster [DBG] pgmap v510: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:42.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:39.679199+0000 mgr.smithi078.zidbgq (mgr.14162) 542 : cluster [DBG] pgmap v510: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:43.032 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:43.271 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:43.767 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:41.679625+0000 mgr.smithi078.zidbgq (mgr.14162) 543 : cluster [DBG] pgmap v511: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:44.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:41.679625+0000 mgr.smithi078.zidbgq (mgr.14162) 543 : cluster [DBG] pgmap v511: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:44.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.285+0000 7efd11213700 1 -- 172.21.15.78:0/1789628090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd0c0f56c0 msgr2=0x7efd0c0f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:44.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.285+0000 7efd11213700 1 --2- 172.21.15.78:0/1789628090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd0c0f56c0 0x7efd0c0f5ae0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7efd00004660 tx=0x7efd00009b30).stop 2023-12-21T03:55:44.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.285+0000 7efd11213700 1 -- 172.21.15.78:0/1789628090 shutdown_connections 2023-12-21T03:55:44.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.285+0000 7efd11213700 1 --2- 172.21.15.78:0/1789628090 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd0c0f6960 0x7efd0c0f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:44.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.285+0000 7efd11213700 1 --2- 172.21.15.78:0/1789628090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd0c0f56c0 0x7efd0c0f5ae0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.285+0000 7efd11213700 1 -- 172.21.15.78:0/1789628090 >> 172.21.15.78:0/1789628090 conn(0x7efd0c0f0e20 msgr2=0x7efd0c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.285+0000 7efd11213700 1 -- 172.21.15.78:0/1789628090 shutdown_connections 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.285+0000 7efd11213700 1 -- 172.21.15.78:0/1789628090 wait complete. 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd11213700 1 Processor -- start 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd11213700 1 -- start start 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd11213700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd0c0f56c0 0x7efd0c109df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd11213700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd0c0f6960 0x7efd0c10c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd11213700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c0609b0 con 0x7efd0c0f56c0 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd11213700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c060af0 con 0x7efd0c0f6960 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd0ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd0c0f56c0 0x7efd0c109df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd0ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd0c0f56c0 0x7efd0c109df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44994/0 (socket says 172.21.15.78:44994) 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd0ad9d700 1 -- 172.21.15.78:0/2142498703 learned_addr learned my addr 172.21.15.78:0/2142498703 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd0ad9d700 1 -- 172.21.15.78:0/2142498703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd0c0f6960 msgr2=0x7efd0c10c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd0a59c700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd0c0f6960 0x7efd0c10c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd0ad9d700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd0c0f6960 0x7efd0c10c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd0ad9d700 1 -- 172.21.15.78:0/2142498703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd00005040 con 0x7efd0c0f56c0 2023-12-21T03:55:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.286+0000 7efd0a59c700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd0c0f6960 0x7efd0c10c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2023-12-21T03:55:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.287+0000 7efd0ad9d700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd0c0f56c0 0x7efd0c109df0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7efd00009f50 tx=0x7efd00009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.287+0000 7efcf37fe700 1 -- 172.21.15.78:0/2142498703 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd00018070 con 0x7efd0c0f56c0 2023-12-21T03:55:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.287+0000 7efd11213700 1 -- 172.21.15.78:0/2142498703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c10c8a0 con 0x7efd0c0f56c0 2023-12-21T03:55:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.287+0000 7efcf37fe700 1 -- 172.21.15.78:0/2142498703 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd00013410 con 0x7efd0c0f56c0 2023-12-21T03:55:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.287+0000 7efcf37fe700 1 -- 172.21.15.78:0/2142498703 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd0001c620 con 0x7efd0c0f56c0 2023-12-21T03:55:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.287+0000 7efd11213700 1 -- 172.21.15.78:0/2142498703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c10ce00 con 0x7efd0c0f56c0 2023-12-21T03:55:44.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.288+0000 7efd11213700 1 -- 172.21.15.78:0/2142498703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0c045ad0 con 0x7efd0c0f56c0 2023-12-21T03:55:44.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.288+0000 7efcf37fe700 1 -- 172.21.15.78:0/2142498703 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efd0001d420 con 0x7efd0c0f56c0 2023-12-21T03:55:44.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.288+0000 7efcf37fe700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efcf405c2a0 0x7efcf405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:44.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.288+0000 7efcf37fe700 1 -- 172.21.15.78:0/2142498703 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7efd000779a0 con 0x7efd0c0f56c0 2023-12-21T03:55:44.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.288+0000 7efd0a59c700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efcf405c2a0 0x7efcf405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:44.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.289+0000 7efd0a59c700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efcf405c2a0 0x7efcf405e760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7efcfc00a7e0 tx=0x7efcfc008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:44.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.292+0000 7efcf37fe700 1 -- 172.21.15.78:0/2142498703 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd0004a9c0 con 0x7efd0c0f56c0 2023-12-21T03:55:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.499+0000 7efd11213700 1 -- 172.21.15.78:0/2142498703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efd0c027a00 con 0x7efd0c0f56c0 2023-12-21T03:55:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.500+0000 7efcf37fe700 1 -- 172.21.15.78:0/2142498703 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7efd0004dfe0 con 0x7efd0c0f56c0 2023-12-21T03:55:44.501 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.501+0000 7efd11213700 1 -- 172.21.15.78:0/2142498703 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efcf405c2a0 msgr2=0x7efcf405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.501+0000 7efd11213700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efcf405c2a0 0x7efcf405e760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7efcfc00a7e0 tx=0x7efcfc008040).stop 2023-12-21T03:55:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.501+0000 7efd11213700 1 -- 172.21.15.78:0/2142498703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd0c0f56c0 msgr2=0x7efd0c109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.501+0000 7efd11213700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd0c0f56c0 0x7efd0c109df0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7efd00009f50 tx=0x7efd00009b30).stop 2023-12-21T03:55:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.501+0000 7efd11213700 1 -- 172.21.15.78:0/2142498703 shutdown_connections 2023-12-21T03:55:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.501+0000 7efd11213700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7efcf405c2a0 0x7efcf405e760 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.501+0000 7efd11213700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd0c0f56c0 0x7efd0c109df0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.501+0000 7efd11213700 1 --2- 172.21.15.78:0/2142498703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd0c0f6960 0x7efd0c10c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.501+0000 7efd11213700 1 -- 172.21.15.78:0/2142498703 >> 172.21.15.78:0/2142498703 conn(0x7efd0c0f0e20 msgr2=0x7efd0c0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.502+0000 7efd11213700 1 -- 172.21.15.78:0/2142498703 shutdown_connections 2023-12-21T03:55:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:44.502+0000 7efd11213700 1 -- 172.21.15.78:0/2142498703 wait complete. 2023-12-21T03:55:44.911 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:43.679905+0000 mgr.smithi078.zidbgq (mgr.14162) 544 : cluster [DBG] pgmap v512: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:44.501209+0000 mon.smithi078 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.78:0/2142498703' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:43.679905+0000 mgr.smithi078.zidbgq (mgr.14162) 544 : cluster [DBG] pgmap v512: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:44.501209+0000 mon.smithi078 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.78:0/2142498703' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:45.680313+0000 mgr.smithi078.zidbgq (mgr.14162) 545 : cluster [DBG] pgmap v513: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:47.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:45.680313+0000 mgr.smithi078.zidbgq (mgr.14162) 545 : cluster [DBG] pgmap v513: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:47.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:48.155 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:49.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.135+0000 7f0116cf2700 1 -- 172.21.15.78:0/1573883275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01100f67e0 msgr2=0x7f01100f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:49.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.135+0000 7f0116cf2700 1 --2- 172.21.15.78:0/1573883275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01100f67e0 0x7f01100f6c00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0104004660 tx=0x7f0104009b30).stop 2023-12-21T03:55:49.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.135+0000 7f0116cf2700 1 -- 172.21.15.78:0/1573883275 shutdown_connections 2023-12-21T03:55:49.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.135+0000 7f0116cf2700 1 --2- 172.21.15.78:0/1573883275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01100f7a80 0x7f01100f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.135+0000 7f0116cf2700 1 --2- 172.21.15.78:0/1573883275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01100f67e0 0x7f01100f6c00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.135+0000 7f0116cf2700 1 -- 172.21.15.78:0/1573883275 >> 172.21.15.78:0/1573883275 conn(0x7f01100f1f80 msgr2=0x7f01100f43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.135+0000 7f0116cf2700 1 -- 172.21.15.78:0/1573883275 shutdown_connections 2023-12-21T03:55:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.135+0000 7f0116cf2700 1 -- 172.21.15.78:0/1573883275 wait complete. 2023-12-21T03:55:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.135+0000 7f0116cf2700 1 Processor -- start 2023-12-21T03:55:49.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f0116cf2700 1 -- start start 2023-12-21T03:55:49.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f0116cf2700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01100f67e0 0x7f01101070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:49.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f0116cf2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01100f7a80 0x7f01101075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:49.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f0116cf2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0110107be0 con 0x7f01100f7a80 2023-12-21T03:55:49.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f0116cf2700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0110108490 con 0x7f01100f67e0 2023-12-21T03:55:49.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f010ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01100f7a80 0x7f01101075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:49.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f010ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01100f7a80 0x7f01101075a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47894/0 (socket says 172.21.15.78:47894) 2023-12-21T03:55:49.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f010ffff700 1 -- 172.21.15.78:0/3654151218 learned_addr learned my addr 172.21.15.78:0/3654151218 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:49.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f0114a8e700 1 --2- 172.21.15.78:0/3654151218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01100f67e0 0x7f01101070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:49.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f010ffff700 1 -- 172.21.15.78:0/3654151218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01100f67e0 msgr2=0x7f01101070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:49.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f010ffff700 1 --2- 172.21.15.78:0/3654151218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01100f67e0 0x7f01101070b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:49.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f010ffff700 1 -- 172.21.15.78:0/3654151218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0104005040 con 0x7f01100f7a80 2023-12-21T03:55:49.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.136+0000 7f010ffff700 1 --2- 172.21.15.78:0/3654151218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01100f7a80 0x7f01101075a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f00fc00b410 tx=0x7f00fc006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:49.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.137+0000 7f010d7fa700 1 -- 172.21.15.78:0/3654151218 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00fc0125f0 con 0x7f01100f7a80 2023-12-21T03:55:49.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.137+0000 7f0116cf2700 1 -- 172.21.15.78:0/3654151218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0110108770 con 0x7f01100f7a80 2023-12-21T03:55:49.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.137+0000 7f010d7fa700 1 -- 172.21.15.78:0/3654151218 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00fc00be30 con 0x7f01100f7a80 2023-12-21T03:55:49.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.137+0000 7f010d7fa700 1 -- 172.21.15.78:0/3654151218 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00fc002a20 con 0x7f01100f7a80 2023-12-21T03:55:49.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.137+0000 7f0116cf2700 1 -- 172.21.15.78:0/3654151218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0110108d30 con 0x7f01100f7a80 2023-12-21T03:55:49.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.138+0000 7f010d7fa700 1 -- 172.21.15.78:0/3654151218 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f00fc00f030 con 0x7f01100f7a80 2023-12-21T03:55:49.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.138+0000 7f0116cf2700 1 -- 172.21.15.78:0/3654151218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0110059100 con 0x7f01100f7a80 2023-12-21T03:55:49.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.138+0000 7f010d7fa700 1 --2- 172.21.15.78:0/3654151218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f010005c2a0 0x7f010005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:49.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.138+0000 7f010d7fa700 1 -- 172.21.15.78:0/3654151218 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f00fc075ff0 con 0x7f01100f7a80 2023-12-21T03:55:49.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.138+0000 7f0114a8e700 1 --2- 172.21.15.78:0/3654151218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f010005c2a0 0x7f010005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:49.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.139+0000 7f0114a8e700 1 --2- 172.21.15.78:0/3654151218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f010005c2a0 0x7f010005e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f0104004630 tx=0x7f0104009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:49.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.142+0000 7f010d7fa700 1 -- 172.21.15.78:0/3654151218 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00fc048f50 con 0x7f01100f7a80 2023-12-21T03:55:49.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.347+0000 7f0116cf2700 1 -- 172.21.15.78:0/3654151218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f01100feff0 con 0x7f01100f7a80 2023-12-21T03:55:49.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.348+0000 7f010d7fa700 1 -- 172.21.15.78:0/3654151218 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f00fc04c570 con 0x7f01100f7a80 2023-12-21T03:55:49.349 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.350+0000 7f0116cf2700 1 -- 172.21.15.78:0/3654151218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f010005c2a0 msgr2=0x7f010005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.350+0000 7f0116cf2700 1 --2- 172.21.15.78:0/3654151218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f010005c2a0 0x7f010005e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f0104004630 tx=0x7f0104009b30).stop 2023-12-21T03:55:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.350+0000 7f0116cf2700 1 -- 172.21.15.78:0/3654151218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01100f7a80 msgr2=0x7f01101075a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.350+0000 7f0116cf2700 1 --2- 172.21.15.78:0/3654151218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01100f7a80 0x7f01101075a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f00fc00b410 tx=0x7f00fc006ef0).stop 2023-12-21T03:55:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.350+0000 7f0116cf2700 1 -- 172.21.15.78:0/3654151218 shutdown_connections 2023-12-21T03:55:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.350+0000 7f0116cf2700 1 --2- 172.21.15.78:0/3654151218 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f010005c2a0 0x7f010005e760 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.350+0000 7f0116cf2700 1 --2- 172.21.15.78:0/3654151218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01100f67e0 0x7f01101070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.350+0000 7f0116cf2700 1 --2- 172.21.15.78:0/3654151218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01100f7a80 0x7f01101075a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.350+0000 7f0116cf2700 1 -- 172.21.15.78:0/3654151218 >> 172.21.15.78:0/3654151218 conn(0x7f01100f1f80 msgr2=0x7f01100fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.350+0000 7f0116cf2700 1 -- 172.21.15.78:0/3654151218 shutdown_connections 2023-12-21T03:55:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:49.350+0000 7f0116cf2700 1 -- 172.21.15.78:0/3654151218 wait complete. 2023-12-21T03:55:49.736 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:47.680821+0000 mgr.smithi078.zidbgq (mgr.14162) 546 : cluster [DBG] pgmap v514: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:49.737 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:49.349323+0000 mon.smithi078 (mon.0) 556 : audit [DBG] from='client.? 172.21.15.78:0/3654151218' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:47.680821+0000 mgr.smithi078.zidbgq (mgr.14162) 546 : cluster [DBG] pgmap v514: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:49.349323+0000 mon.smithi078 (mon.0) 556 : audit [DBG] from='client.? 172.21.15.78:0/3654151218' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:49.768 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:49.681226+0000 mgr.smithi078.zidbgq (mgr.14162) 547 : cluster [DBG] pgmap v515: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:51.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:49.681226+0000 mgr.smithi078.zidbgq (mgr.14162) 547 : cluster [DBG] pgmap v515: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:52.769 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:53.006 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:53.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:51.681710+0000 mgr.smithi078.zidbgq (mgr.14162) 548 : cluster [DBG] pgmap v516: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:51.681710+0000 mgr.smithi078.zidbgq (mgr.14162) 548 : cluster [DBG] pgmap v516: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:54.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.028+0000 7fce8e8ba700 1 -- 172.21.15.78:0/4273592713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce880f99b0 msgr2=0x7fce880fbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:54.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.028+0000 7fce8e8ba700 1 --2- 172.21.15.78:0/4273592713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce880f99b0 0x7fce880fbda0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fce70004660 tx=0x7fce70009b30).stop 2023-12-21T03:55:54.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.028+0000 7fce8e8ba700 1 -- 172.21.15.78:0/4273592713 shutdown_connections 2023-12-21T03:55:54.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.028+0000 7fce8e8ba700 1 --2- 172.21.15.78:0/4273592713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce880f99b0 0x7fce880fbda0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:54.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.028+0000 7fce8e8ba700 1 --2- 172.21.15.78:0/4273592713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce88060810 0x7fce880f9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:54.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.028+0000 7fce8e8ba700 1 -- 172.21.15.78:0/4273592713 >> 172.21.15.78:0/4273592713 conn(0x7fce880f0e50 msgr2=0x7fce880f3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:54.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.028+0000 7fce8e8ba700 1 -- 172.21.15.78:0/4273592713 shutdown_connections 2023-12-21T03:55:54.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.028+0000 7fce8e8ba700 1 -- 172.21.15.78:0/4273592713 wait complete. 2023-12-21T03:55:54.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce8e8ba700 1 Processor -- start 2023-12-21T03:55:54.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce8e8ba700 1 -- start start 2023-12-21T03:55:54.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce8e8ba700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce88060810 0x7fce88107040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:54.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce8e8ba700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce880f99b0 0x7fce88107530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:54.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce8e8ba700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce88107b70 con 0x7fce88060810 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce8e8ba700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce88108420 con 0x7fce880f99b0 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce87fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce88060810 0x7fce88107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce87fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce88060810 0x7fce88107040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47910/0 (socket says 172.21.15.78:47910) 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce87fff700 1 -- 172.21.15.78:0/1604226142 learned_addr learned my addr 172.21.15.78:0/1604226142 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce877fe700 1 --2- 172.21.15.78:0/1604226142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce880f99b0 0x7fce88107530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce87fff700 1 -- 172.21.15.78:0/1604226142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce880f99b0 msgr2=0x7fce88107530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce87fff700 1 --2- 172.21.15.78:0/1604226142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce880f99b0 0x7fce88107530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce87fff700 1 -- 172.21.15.78:0/1604226142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce70005040 con 0x7fce88060810 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.029+0000 7fce87fff700 1 --2- 172.21.15.78:0/1604226142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce88060810 0x7fce88107040 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fce7800b410 tx=0x7fce78006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.030+0000 7fce84ff9700 1 -- 172.21.15.78:0/1604226142 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce780125f0 con 0x7fce88060810 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.030+0000 7fce84ff9700 1 -- 172.21.15.78:0/1604226142 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce78012c00 con 0x7fce88060810 2023-12-21T03:55:54.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.030+0000 7fce84ff9700 1 -- 172.21.15.78:0/1604226142 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce78002aa0 con 0x7fce88060810 2023-12-21T03:55:54.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.030+0000 7fce8e8ba700 1 -- 172.21.15.78:0/1604226142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce88108700 con 0x7fce88060810 2023-12-21T03:55:54.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.030+0000 7fce8e8ba700 1 -- 172.21.15.78:0/1604226142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce88108cc0 con 0x7fce88060810 2023-12-21T03:55:54.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.031+0000 7fce8e8ba700 1 -- 172.21.15.78:0/1604226142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce88059100 con 0x7fce88060810 2023-12-21T03:55:54.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.031+0000 7fce84ff9700 1 -- 172.21.15.78:0/1604226142 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fce7800f030 con 0x7fce88060810 2023-12-21T03:55:54.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.031+0000 7fce84ff9700 1 --2- 172.21.15.78:0/1604226142 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fce7405c2a0 0x7fce7405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:54.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.031+0000 7fce84ff9700 1 -- 172.21.15.78:0/1604226142 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fce78076060 con 0x7fce88060810 2023-12-21T03:55:54.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.031+0000 7fce877fe700 1 --2- 172.21.15.78:0/1604226142 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fce7405c2a0 0x7fce7405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:54.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.032+0000 7fce877fe700 1 --2- 172.21.15.78:0/1604226142 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fce7405c2a0 0x7fce7405e760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fce70004e30 tx=0x7fce70009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:54.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.035+0000 7fce84ff9700 1 -- 172.21.15.78:0/1604226142 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce78049000 con 0x7fce88060810 2023-12-21T03:55:54.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.244+0000 7fce8e8ba700 1 -- 172.21.15.78:0/1604226142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fce88045ad0 con 0x7fce88060810 2023-12-21T03:55:54.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.245+0000 7fce84ff9700 1 -- 172.21.15.78:0/1604226142 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fce7804c620 con 0x7fce88060810 2023-12-21T03:55:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:54.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.247+0000 7fce8e8ba700 1 -- 172.21.15.78:0/1604226142 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fce7405c2a0 msgr2=0x7fce7405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:54.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.247+0000 7fce8e8ba700 1 --2- 172.21.15.78:0/1604226142 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fce7405c2a0 0x7fce7405e760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fce70004e30 tx=0x7fce70009b30).stop 2023-12-21T03:55:54.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.247+0000 7fce8e8ba700 1 -- 172.21.15.78:0/1604226142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce88060810 msgr2=0x7fce88107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:54.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.247+0000 7fce8e8ba700 1 --2- 172.21.15.78:0/1604226142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce88060810 0x7fce88107040 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fce7800b410 tx=0x7fce78006ef0).stop 2023-12-21T03:55:54.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.247+0000 7fce8e8ba700 1 -- 172.21.15.78:0/1604226142 shutdown_connections 2023-12-21T03:55:54.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.247+0000 7fce8e8ba700 1 --2- 172.21.15.78:0/1604226142 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fce7405c2a0 0x7fce7405e760 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:54.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.247+0000 7fce8e8ba700 1 --2- 172.21.15.78:0/1604226142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce88060810 0x7fce88107040 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:54.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.247+0000 7fce8e8ba700 1 --2- 172.21.15.78:0/1604226142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce880f99b0 0x7fce88107530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:54.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.247+0000 7fce8e8ba700 1 -- 172.21.15.78:0/1604226142 >> 172.21.15.78:0/1604226142 conn(0x7fce880f0e50 msgr2=0x7fce880f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:54.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.247+0000 7fce8e8ba700 1 -- 172.21.15.78:0/1604226142 shutdown_connections 2023-12-21T03:55:54.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:54.247+0000 7fce8e8ba700 1 -- 172.21.15.78:0/1604226142 wait complete. 2023-12-21T03:55:54.552 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:54 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:54.246278+0000 mon.smithi078 (mon.0) 557 : audit [DBG] from='client.? 172.21.15.78:0/1604226142' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:54.619 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:54.762 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:54 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:54.246278+0000 mon.smithi078 (mon.0) 557 : audit [DBG] from='client.? 172.21.15.78:0/1604226142' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:53.682067+0000 mgr.smithi078.zidbgq (mgr.14162) 549 : cluster [DBG] pgmap v517: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:53.682067+0000 mgr.smithi078.zidbgq (mgr.14162) 549 : cluster [DBG] pgmap v517: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:57.620 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:55:57.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:55.682453+0000 mgr.smithi078.zidbgq (mgr.14162) 550 : cluster [DBG] pgmap v518: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:57.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:55.682453+0000 mgr.smithi078.zidbgq (mgr.14162) 550 : cluster [DBG] pgmap v518: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:57.855 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:55:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.816+0000 7fad837b1700 1 -- 172.21.15.78:0/3498864814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad7c0f56f0 msgr2=0x7fad7c0f5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.816+0000 7fad837b1700 1 --2- 172.21.15.78:0/3498864814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad7c0f56f0 0x7fad7c0f5b10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fad6c004660 tx=0x7fad6c009b30).stop 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.816+0000 7fad837b1700 1 -- 172.21.15.78:0/3498864814 shutdown_connections 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.816+0000 7fad837b1700 1 --2- 172.21.15.78:0/3498864814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad7c0f6990 0x7fad7c0f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.816+0000 7fad837b1700 1 --2- 172.21.15.78:0/3498864814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad7c0f56f0 0x7fad7c0f5b10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.816+0000 7fad837b1700 1 -- 172.21.15.78:0/3498864814 >> 172.21.15.78:0/3498864814 conn(0x7fad7c0f0e50 msgr2=0x7fad7c0f3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.816+0000 7fad837b1700 1 -- 172.21.15.78:0/3498864814 shutdown_connections 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.816+0000 7fad837b1700 1 -- 172.21.15.78:0/3498864814 wait complete. 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.816+0000 7fad837b1700 1 Processor -- start 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.816+0000 7fad837b1700 1 -- start start 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad837b1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad7c0f56f0 0x7fad7c1092b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad837b1700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad7c0f6990 0x7fad7c1097a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad837b1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c109de0 con 0x7fad7c0f56f0 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad837b1700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c10a690 con 0x7fad7c0f6990 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad8154d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad7c0f56f0 0x7fad7c1092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:58.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad8154d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad7c0f56f0 0x7fad7c1092b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60478/0 (socket says 172.21.15.78:60478) 2023-12-21T03:55:58.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad8154d700 1 -- 172.21.15.78:0/2264505245 learned_addr learned my addr 172.21.15.78:0/2264505245 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:55:58.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad8154d700 1 -- 172.21.15.78:0/2264505245 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad7c0f6990 msgr2=0x7fad7c1097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:58.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad8154d700 1 --2- 172.21.15.78:0/2264505245 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad7c0f6990 0x7fad7c1097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:58.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad8154d700 1 -- 172.21.15.78:0/2264505245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c005040 con 0x7fad7c0f56f0 2023-12-21T03:55:58.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad8154d700 1 --2- 172.21.15.78:0/2264505245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad7c0f56f0 0x7fad7c1092b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fad6c009f50 tx=0x7fad6c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:58.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad71ffb700 1 -- 172.21.15.78:0/2264505245 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad6c018070 con 0x7fad7c0f56f0 2023-12-21T03:55:58.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad837b1700 1 -- 172.21.15.78:0/2264505245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7c10a910 con 0x7fad7c0f56f0 2023-12-21T03:55:58.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad71ffb700 1 -- 172.21.15.78:0/2264505245 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad6c013410 con 0x7fad7c0f56f0 2023-12-21T03:55:58.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad71ffb700 1 -- 172.21.15.78:0/2264505245 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad6c01c620 con 0x7fad7c0f56f0 2023-12-21T03:55:58.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.817+0000 7fad837b1700 1 -- 172.21.15.78:0/2264505245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7c10ae70 con 0x7fad7c0f56f0 2023-12-21T03:55:58.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.818+0000 7fad837b1700 1 -- 172.21.15.78:0/2264505245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7c059100 con 0x7fad7c0f56f0 2023-12-21T03:55:58.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.819+0000 7fad71ffb700 1 -- 172.21.15.78:0/2264505245 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fad6c01d420 con 0x7fad7c0f56f0 2023-12-21T03:55:58.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.819+0000 7fad71ffb700 1 --2- 172.21.15.78:0/2264505245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad6805c2a0 0x7fad6805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:55:58.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.819+0000 7fad71ffb700 1 -- 172.21.15.78:0/2264505245 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fad6c077eb0 con 0x7fad7c0f56f0 2023-12-21T03:55:58.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.819+0000 7fad80d4c700 1 --2- 172.21.15.78:0/2264505245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad6805c2a0 0x7fad6805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:55:58.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.820+0000 7fad80d4c700 1 --2- 172.21.15.78:0/2264505245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad6805c2a0 0x7fad6805e760 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fad78009610 tx=0x7fad78006ab0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:55:58.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:58.823+0000 7fad71ffb700 1 -- 172.21.15.78:0/2264505245 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad6c04afa0 con 0x7fad7c0f56f0 2023-12-21T03:55:59.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.028+0000 7fad837b1700 1 -- 172.21.15.78:0/2264505245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fad7c045ad0 con 0x7fad7c0f56f0 2023-12-21T03:55:59.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.028+0000 7fad71ffb700 1 -- 172.21.15.78:0/2264505245 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fad6c04e5c0 con 0x7fad7c0f56f0 2023-12-21T03:55:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:55:59.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.030+0000 7fad837b1700 1 -- 172.21.15.78:0/2264505245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad6805c2a0 msgr2=0x7fad6805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:59.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.030+0000 7fad837b1700 1 --2- 172.21.15.78:0/2264505245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad6805c2a0 0x7fad6805e760 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fad78009610 tx=0x7fad78006ab0).stop 2023-12-21T03:55:59.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.030+0000 7fad837b1700 1 -- 172.21.15.78:0/2264505245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad7c0f56f0 msgr2=0x7fad7c1092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:55:59.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.030+0000 7fad837b1700 1 --2- 172.21.15.78:0/2264505245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad7c0f56f0 0x7fad7c1092b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fad6c009f50 tx=0x7fad6c009b30).stop 2023-12-21T03:55:59.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.030+0000 7fad837b1700 1 -- 172.21.15.78:0/2264505245 shutdown_connections 2023-12-21T03:55:59.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.030+0000 7fad837b1700 1 --2- 172.21.15.78:0/2264505245 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad6805c2a0 0x7fad6805e760 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:59.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.030+0000 7fad837b1700 1 --2- 172.21.15.78:0/2264505245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad7c0f56f0 0x7fad7c1092b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:55:59.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.030+0000 7fad837b1700 1 --2- 172.21.15.78:0/2264505245 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad7c0f6990 0x7fad7c1097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:55:59.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.030+0000 7fad837b1700 1 -- 172.21.15.78:0/2264505245 >> 172.21.15.78:0/2264505245 conn(0x7fad7c0f0e50 msgr2=0x7fad7c0603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:55:59.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.030+0000 7fad837b1700 1 -- 172.21.15.78:0/2264505245 shutdown_connections 2023-12-21T03:55:59.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:55:59.030+0000 7fad837b1700 1 -- 172.21.15.78:0/2264505245 wait complete. 2023-12-21T03:55:59.418 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:55:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:57.682818+0000 mgr.smithi078.zidbgq (mgr.14162) 551 : cluster [DBG] pgmap v519: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:59.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:55:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:55:59.029794+0000 mon.smithi078 (mon.0) 558 : audit [DBG] from='client.? 172.21.15.78:0/2264505245' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:55:59.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:57.682818+0000 mgr.smithi078.zidbgq (mgr.14162) 551 : cluster [DBG] pgmap v519: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:55:59.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:55:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:55:59.029794+0000 mon.smithi078 (mon.0) 558 : audit [DBG] from='client.? 172.21.15.78:0/2264505245' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:55:59.683124+0000 mgr.smithi078.zidbgq (mgr.14162) 552 : cluster [DBG] pgmap v520: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:01.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:55:59.683124+0000 mgr.smithi078.zidbgq (mgr.14162) 552 : cluster [DBG] pgmap v520: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:02.419 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:02.658 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.654+0000 7fb55d630700 1 -- 172.21.15.78:0/219623841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5580f5540 msgr2=0x7fb5580f5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.654+0000 7fb55d630700 1 --2- 172.21.15.78:0/219623841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5580f5540 0x7fb5580f5960 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fb548004660 tx=0x7fb548009b30).stop 2023-12-21T03:56:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.654+0000 7fb55d630700 1 -- 172.21.15.78:0/219623841 shutdown_connections 2023-12-21T03:56:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.654+0000 7fb55d630700 1 --2- 172.21.15.78:0/219623841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5580f67e0 0x7fb5580f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.654+0000 7fb55d630700 1 --2- 172.21.15.78:0/219623841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5580f5540 0x7fb5580f5960 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.654+0000 7fb55d630700 1 -- 172.21.15.78:0/219623841 >> 172.21.15.78:0/219623841 conn(0x7fb5580f0ca0 msgr2=0x7fb5580f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.655+0000 7fb55d630700 1 -- 172.21.15.78:0/219623841 shutdown_connections 2023-12-21T03:56:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.655+0000 7fb55d630700 1 -- 172.21.15.78:0/219623841 wait complete. 2023-12-21T03:56:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.655+0000 7fb55d630700 1 Processor -- start 2023-12-21T03:56:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.655+0000 7fb55d630700 1 -- start start 2023-12-21T03:56:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb55d630700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5580f5540 0x7fb558109c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb55d630700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5580f67e0 0x7fb55810c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb55d630700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb558060920 con 0x7fb5580f67e0 2023-12-21T03:56:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb55d630700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb558060a60 con 0x7fb5580f5540 2023-12-21T03:56:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb5567fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5580f67e0 0x7fb55810c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb5567fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5580f67e0 0x7fb55810c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60492/0 (socket says 172.21.15.78:60492) 2023-12-21T03:56:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb5567fc700 1 -- 172.21.15.78:0/778439498 learned_addr learned my addr 172.21.15.78:0/778439498 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb556ffd700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5580f5540 0x7fb558109c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb5567fc700 1 -- 172.21.15.78:0/778439498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5580f5540 msgr2=0x7fb558109c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb5567fc700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5580f5540 0x7fb558109c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb5567fc700 1 -- 172.21.15.78:0/778439498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548005040 con 0x7fb5580f67e0 2023-12-21T03:56:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb556ffd700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5580f5540 0x7fb558109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2023-12-21T03:56:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb5567fc700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5580f67e0 0x7fb55810c130 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fb54c0026d0 tx=0x7fb54c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.656+0000 7fb53f7fe700 1 -- 172.21.15.78:0/778439498 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb54c0079c0 con 0x7fb5580f67e0 2023-12-21T03:56:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.657+0000 7fb55d630700 1 -- 172.21.15.78:0/778439498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55810c6e0 con 0x7fb5580f67e0 2023-12-21T03:56:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.657+0000 7fb53f7fe700 1 -- 172.21.15.78:0/778439498 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb54c00f040 con 0x7fb5580f67e0 2023-12-21T03:56:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.657+0000 7fb53f7fe700 1 -- 172.21.15.78:0/778439498 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb54c0148b0 con 0x7fb5580f67e0 2023-12-21T03:56:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.657+0000 7fb55d630700 1 -- 172.21.15.78:0/778439498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55810cc70 con 0x7fb5580f67e0 2023-12-21T03:56:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.658+0000 7fb55d630700 1 -- 172.21.15.78:0/778439498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb558045ad0 con 0x7fb5580f67e0 2023-12-21T03:56:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.659+0000 7fb53f7fe700 1 -- 172.21.15.78:0/778439498 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb54c007b20 con 0x7fb5580f67e0 2023-12-21T03:56:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.659+0000 7fb53f7fe700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb54005c2b0 0x7fb54005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.659+0000 7fb53f7fe700 1 -- 172.21.15.78:0/778439498 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb54c075ef0 con 0x7fb5580f67e0 2023-12-21T03:56:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.659+0000 7fb556ffd700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb54005c2b0 0x7fb54005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.659+0000 7fb556ffd700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb54005c2b0 0x7fb54005e770 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fb5580605a0 tx=0x7fb548009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:03.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.663+0000 7fb53f7fe700 1 -- 172.21.15.78:0/778439498 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb54c048fe0 con 0x7fb5580f67e0 2023-12-21T03:56:03.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:01.683538+0000 mgr.smithi078.zidbgq (mgr.14162) 553 : cluster [DBG] pgmap v521: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:03.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.306157+0000 mon.smithi078 (mon.0) 559 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-21T03:56:03.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.307234+0000 mon.smithi078 (mon.0) 560 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.78:9093"}]: dispatch 2023-12-21T03:56:03.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.311566+0000 mon.smithi078 (mon.0) 561 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:03.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.315121+0000 mon.smithi078 (mon.0) 562 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-21T03:56:03.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.319957+0000 mon.smithi078 (mon.0) 563 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.78:3000"}]: dispatch 2023-12-21T03:56:03.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.326592+0000 mon.smithi078 (mon.0) 564 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:03.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.328744+0000 mon.smithi078 (mon.0) 565 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:56:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:01.683538+0000 mgr.smithi078.zidbgq (mgr.14162) 553 : cluster [DBG] pgmap v521: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.306157+0000 mon.smithi078 (mon.0) 559 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-21T03:56:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.307234+0000 mon.smithi078 (mon.0) 560 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.78:9093"}]: dispatch 2023-12-21T03:56:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.311566+0000 mon.smithi078 (mon.0) 561 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.315121+0000 mon.smithi078 (mon.0) 562 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-21T03:56:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.319957+0000 mon.smithi078 (mon.0) 563 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.78:3000"}]: dispatch 2023-12-21T03:56:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.326592+0000 mon.smithi078 (mon.0) 564 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:03.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.328744+0000 mon.smithi078 (mon.0) 565 : audit [DBG] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-21T03:56:03.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.896+0000 7fb55d630700 1 -- 172.21.15.78:0/778439498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb5580fde50 con 0x7fb5580f67e0 2023-12-21T03:56:03.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.896+0000 7fb53f7fe700 1 -- 172.21.15.78:0/778439498 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fb54c04c600 con 0x7fb5580f67e0 2023-12-21T03:56:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.898+0000 7fb55d630700 1 -- 172.21.15.78:0/778439498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb54005c2b0 msgr2=0x7fb54005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.898+0000 7fb55d630700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb54005c2b0 0x7fb54005e770 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fb5580605a0 tx=0x7fb548009b30).stop 2023-12-21T03:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.898+0000 7fb55d630700 1 -- 172.21.15.78:0/778439498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5580f67e0 msgr2=0x7fb55810c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.898+0000 7fb55d630700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5580f67e0 0x7fb55810c130 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fb54c0026d0 tx=0x7fb54c009de0).stop 2023-12-21T03:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.898+0000 7fb55d630700 1 -- 172.21.15.78:0/778439498 shutdown_connections 2023-12-21T03:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.898+0000 7fb55d630700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb54005c2b0 0x7fb54005e770 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.898+0000 7fb55d630700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5580f5540 0x7fb558109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.898+0000 7fb55d630700 1 --2- 172.21.15.78:0/778439498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5580f67e0 0x7fb55810c130 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.898+0000 7fb55d630700 1 -- 172.21.15.78:0/778439498 >> 172.21.15.78:0/778439498 conn(0x7fb5580f0ca0 msgr2=0x7fb5580fa230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.898+0000 7fb55d630700 1 -- 172.21.15.78:0/778439498 shutdown_connections 2023-12-21T03:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:03.898+0000 7fb55d630700 1 -- 172.21.15.78:0/778439498 wait complete. 2023-12-21T03:56:04.372 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:04.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:56:03.304133+0000 mgr.smithi078.zidbgq (mgr.14162) 554 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi148 ... 2023-12-21T03:56:04.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 1 from systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:56:04.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service failed because a timeout was exceeded. 2023-12-21T03:56:04.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service" and "journalctl -xe" for details. 2023-12-21T03:56:04.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:56:04.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:56:04.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:56:04.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:56:04.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ports=daemon_ports) 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: get_unit_name(fsid, daemon_type, daemon_id)]) 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: Failed command: systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: yield (conn, connr) 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:56:04.818 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: code, '\n'.join(err))) 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi148 ... 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Non-zero exit code 1 from systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service failed because a timeout was exceeded. 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service" and "journalctl -xe" for details. 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: Traceback (most recent call last): 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: main() 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: r = ctx.func(ctx) 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: return func(ctx) 2023-12-21T03:56:04.819 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: ports=daemon_ports) 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: get_unit_name(fsid, daemon_type, daemon_id)]) 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: RuntimeError: Failed command: systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.306486+0000 mgr.smithi078.zidbgq (mgr.14162) 555 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.307442+0000 mgr.smithi078.zidbgq (mgr.14162) 556 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.78:9093"}]: dispatch 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.315596+0000 mgr.smithi078.zidbgq (mgr.14162) 557 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.320397+0000 mgr.smithi078.zidbgq (mgr.14162) 558 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.78:3000"}]: dispatch 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.559498+0000 mon.smithi078 (mon.0) 566 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:03.897750+0000 mon.smithi078 (mon.0) 567 : audit [DBG] from='client.? 172.21.15.78:0/778439498' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:04.820 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:04.488061+0000 mon.smithi078 (mon.0) 568 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:04.839 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:56:03.304133+0000 mgr.smithi078.zidbgq (mgr.14162) 554 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi148 ... 2023-12-21T03:56:04.839 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Non-zero exit code 1 from systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:56:04.839 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service failed because a timeout was exceeded. 2023-12-21T03:56:04.839 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service" and "journalctl -xe" for details. 2023-12-21T03:56:04.839 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:56:04.840 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:56:04.840 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: main() 2023-12-21T03:56:04.840 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:56:04.840 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: r = ctx.func(ctx) 2023-12-21T03:56:04.840 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:56:04.840 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: return func(ctx) 2023-12-21T03:56:04.841 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:56:04.841 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: ports=daemon_ports) 2023-12-21T03:56:04.841 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2023-12-21T03:56:04.841 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: get_unit_name(fsid, daemon_type, daemon_id)]) 2023-12-21T03:56:04.841 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:56:04.841 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:56:04.842 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: RuntimeError: Failed command: systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:56:04.842 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:56:04.842 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-21T03:56:04.842 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: yield (conn, connr) 2023-12-21T03:56:04.842 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-21T03:56:04.842 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: code, '\n'.join(err))) 2023-12-21T03:56:04.842 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi148 ... 2023-12-21T03:56:04.843 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Non-zero exit code 1 from systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:56:04.843 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr Job for ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service failed because a timeout was exceeded. 2023-12-21T03:56:04.843 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: systemctl: stderr See "systemctl status ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148.service" and "journalctl -xe" for details. 2023-12-21T03:56:04.843 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: Traceback (most recent call last): 2023-12-21T03:56:04.843 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-21T03:56:04.843 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: main() 2023-12-21T03:56:04.843 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-21T03:56:04.843 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: r = ctx.func(ctx) 2023-12-21T03:56:04.843 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-21T03:56:04.844 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: return func(ctx) 2023-12-21T03:56:04.844 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-21T03:56:04.844 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: ports=daemon_ports) 2023-12-21T03:56:04.844 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2023-12-21T03:56:04.844 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: get_unit_name(fsid, daemon_type, daemon_id)]) 2023-12-21T03:56:04.844 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: File "/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-21T03:56:04.844 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-21T03:56:04.844 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: RuntimeError: Failed command: systemctl restart ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@node-exporter.smithi148 2023-12-21T03:56:04.845 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.306486+0000 mgr.smithi078.zidbgq (mgr.14162) 555 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-21T03:56:04.845 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.307442+0000 mgr.smithi078.zidbgq (mgr.14162) 556 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.78:9093"}]: dispatch 2023-12-21T03:56:04.845 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.315596+0000 mgr.smithi078.zidbgq (mgr.14162) 557 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-21T03:56:04.845 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.320397+0000 mgr.smithi078.zidbgq (mgr.14162) 558 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.78:3000"}]: dispatch 2023-12-21T03:56:04.845 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.559498+0000 mon.smithi078 (mon.0) 566 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:04.845 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:03.897750+0000 mon.smithi078 (mon.0) 567 : audit [DBG] from='client.? 172.21.15.78:0/778439498' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:04.846 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:04 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:04.488061+0000 mon.smithi078 (mon.0) 568 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:03.683836+0000 mgr.smithi078.zidbgq (mgr.14162) 559 : cluster [DBG] pgmap v522: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:05.406206+0000 mon.smithi078 (mon.0) 569 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:05.909 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:03.683836+0000 mgr.smithi078.zidbgq (mgr.14162) 559 : cluster [DBG] pgmap v522: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:05.909 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:05.406206+0000 mon.smithi078 (mon.0) 569 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:07.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:07.605 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:07.909 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:05.684241+0000 mgr.smithi078.zidbgq (mgr.14162) 560 : cluster [DBG] pgmap v523: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:07.910 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:06.883087+0000 mon.smithi078 (mon.0) 570 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:08.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:05.684241+0000 mgr.smithi078.zidbgq (mgr.14162) 560 : cluster [DBG] pgmap v523: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:08.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:06.883087+0000 mon.smithi078 (mon.0) 570 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:08.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.557+0000 7fde1094b700 1 -- 172.21.15.78:0/1446830587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0c0f6960 msgr2=0x7fde0c0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:08.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.557+0000 7fde1094b700 1 --2- 172.21.15.78:0/1446830587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0c0f6960 0x7fde0c0f6dc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fde00004660 tx=0x7fde00009b30).stop 2023-12-21T03:56:08.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.557+0000 7fde1094b700 1 -- 172.21.15.78:0/1446830587 shutdown_connections 2023-12-21T03:56:08.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.557+0000 7fde1094b700 1 --2- 172.21.15.78:0/1446830587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0c0f6960 0x7fde0c0f6dc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:08.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.557+0000 7fde1094b700 1 --2- 172.21.15.78:0/1446830587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde0c0f56c0 0x7fde0c0f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:08.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.557+0000 7fde1094b700 1 -- 172.21.15.78:0/1446830587 >> 172.21.15.78:0/1446830587 conn(0x7fde0c0f0e20 msgr2=0x7fde0c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:08.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.557+0000 7fde1094b700 1 -- 172.21.15.78:0/1446830587 shutdown_connections 2023-12-21T03:56:08.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.557+0000 7fde1094b700 1 -- 172.21.15.78:0/1446830587 wait complete. 2023-12-21T03:56:08.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.557+0000 7fde1094b700 1 Processor -- start 2023-12-21T03:56:08.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde1094b700 1 -- start start 2023-12-21T03:56:08.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde1094b700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde0c0f56c0 0x7fde0c109e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:08.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde1094b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0c0f6960 0x7fde0c10c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:08.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde1094b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c060920 con 0x7fde0c0f6960 2023-12-21T03:56:08.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde1094b700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c060a60 con 0x7fde0c0f56c0 2023-12-21T03:56:08.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde09d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0c0f6960 0x7fde0c10c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde09d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0c0f6960 0x7fde0c10c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44844/0 (socket says 172.21.15.78:44844) 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde09d9b700 1 -- 172.21.15.78:0/236323052 learned_addr learned my addr 172.21.15.78:0/236323052 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde0a59c700 1 --2- 172.21.15.78:0/236323052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde0c0f56c0 0x7fde0c109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde09d9b700 1 -- 172.21.15.78:0/236323052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde0c0f56c0 msgr2=0x7fde0c109e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde09d9b700 1 --2- 172.21.15.78:0/236323052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde0c0f56c0 0x7fde0c109e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde09d9b700 1 -- 172.21.15.78:0/236323052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde00005040 con 0x7fde0c0f6960 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde09d9b700 1 --2- 172.21.15.78:0/236323052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0c0f6960 0x7fde0c10c300 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fde00009f50 tx=0x7fde00009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fddfaffd700 1 -- 172.21.15.78:0/236323052 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde00018070 con 0x7fde0c0f6960 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fddfaffd700 1 -- 172.21.15.78:0/236323052 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde00013410 con 0x7fde0c0f6960 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde1094b700 1 -- 172.21.15.78:0/236323052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c10c850 con 0x7fde0c0f6960 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fddfaffd700 1 -- 172.21.15.78:0/236323052 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde0001c3c0 con 0x7fde0c0f6960 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.558+0000 7fde1094b700 1 -- 172.21.15.78:0/236323052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c10cd80 con 0x7fde0c0f6960 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.559+0000 7fde1094b700 1 -- 172.21.15.78:0/236323052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0c059100 con 0x7fde0c0f6960 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.560+0000 7fddfaffd700 1 -- 172.21.15.78:0/236323052 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fde0001d420 con 0x7fde0c0f6960 2023-12-21T03:56:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.560+0000 7fddfaffd700 1 --2- 172.21.15.78:0/236323052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fddf405c2a0 0x7fddf405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.560+0000 7fddfaffd700 1 -- 172.21.15.78:0/236323052 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fde00077800 con 0x7fde0c0f6960 2023-12-21T03:56:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.560+0000 7fde0a59c700 1 --2- 172.21.15.78:0/236323052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fddf405c2a0 0x7fddf405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.560+0000 7fde0a59c700 1 --2- 172.21.15.78:0/236323052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fddf405c2a0 0x7fddf405e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fde0c0605a0 tx=0x7fddfc00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:08.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.564+0000 7fddfaffd700 1 -- 172.21.15.78:0/236323052 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde0004a800 con 0x7fde0c0f6960 2023-12-21T03:56:08.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.773+0000 7fde1094b700 1 -- 172.21.15.78:0/236323052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fde0c045ad0 con 0x7fde0c0f6960 2023-12-21T03:56:08.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.774+0000 7fddfaffd700 1 -- 172.21.15.78:0/236323052 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fde0004de20 con 0x7fde0c0f6960 2023-12-21T03:56:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:08.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.775+0000 7fde1094b700 1 -- 172.21.15.78:0/236323052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fddf405c2a0 msgr2=0x7fddf405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:08.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.775+0000 7fde1094b700 1 --2- 172.21.15.78:0/236323052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fddf405c2a0 0x7fddf405e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fde0c0605a0 tx=0x7fddfc00b040).stop 2023-12-21T03:56:08.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.775+0000 7fde1094b700 1 -- 172.21.15.78:0/236323052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0c0f6960 msgr2=0x7fde0c10c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:08.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.775+0000 7fde1094b700 1 --2- 172.21.15.78:0/236323052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0c0f6960 0x7fde0c10c300 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fde00009f50 tx=0x7fde00009b30).stop 2023-12-21T03:56:08.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.775+0000 7fde1094b700 1 -- 172.21.15.78:0/236323052 shutdown_connections 2023-12-21T03:56:08.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.775+0000 7fde1094b700 1 --2- 172.21.15.78:0/236323052 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fddf405c2a0 0x7fddf405e760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:08.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.775+0000 7fde1094b700 1 --2- 172.21.15.78:0/236323052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde0c0f56c0 0x7fde0c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:08.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.775+0000 7fde1094b700 1 --2- 172.21.15.78:0/236323052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0c0f6960 0x7fde0c10c300 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:08.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.775+0000 7fde1094b700 1 -- 172.21.15.78:0/236323052 >> 172.21.15.78:0/236323052 conn(0x7fde0c0f0e20 msgr2=0x7fde0c0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:08.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.775+0000 7fde1094b700 1 -- 172.21.15.78:0/236323052 shutdown_connections 2023-12-21T03:56:08.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:08.775+0000 7fde1094b700 1 -- 172.21.15.78:0/236323052 wait complete. 2023-12-21T03:56:08.939 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:56:06.876918+0000 mgr.smithi078.zidbgq (mgr.14162) 561 : cephadm [INF] Detected new or changed devices on smithi148 2023-12-21T03:56:08.939 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:07.870715+0000 mon.smithi078 (mon.0) 571 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:08.940 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:08 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:08.480764+0000 mon.smithi078 (mon.0) 572 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:09.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:56:06.876918+0000 mgr.smithi078.zidbgq (mgr.14162) 561 : cephadm [INF] Detected new or changed devices on smithi148 2023-12-21T03:56:09.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:07.870715+0000 mon.smithi078 (mon.0) 571 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:09.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:08.480764+0000 mon.smithi078 (mon.0) 572 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' 2023-12-21T03:56:09.159 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:10.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:07.684547+0000 mgr.smithi078.zidbgq (mgr.14162) 562 : cluster [DBG] pgmap v524: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:10.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cephadm 2023-12-21T03:56:08.484796+0000 mgr.smithi078.zidbgq (mgr.14162) 563 : cephadm [INF] Deploying daemon prometheus.smithi148 on smithi148 2023-12-21T03:56:10.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:08.775064+0000 mon.smithi078 (mon.0) 573 : audit [DBG] from='client.? 172.21.15.78:0/236323052' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:10.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:07.684547+0000 mgr.smithi078.zidbgq (mgr.14162) 562 : cluster [DBG] pgmap v524: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:10.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cephadm 2023-12-21T03:56:08.484796+0000 mgr.smithi078.zidbgq (mgr.14162) 563 : cephadm [INF] Deploying daemon prometheus.smithi148 on smithi148 2023-12-21T03:56:10.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:08.775064+0000 mon.smithi078 (mon.0) 573 : audit [DBG] from='client.? 172.21.15.78:0/236323052' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:12.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:09.684831+0000 mgr.smithi078.zidbgq (mgr.14162) 564 : cluster [DBG] pgmap v525: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:12.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:09.684831+0000 mgr.smithi078.zidbgq (mgr.14162) 564 : cluster [DBG] pgmap v525: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:12.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:13.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.332+0000 7f0039dfc700 1 -- 172.21.15.78:0/714784630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00340f56c0 msgr2=0x7f00340f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:13.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.332+0000 7f0039dfc700 1 --2- 172.21.15.78:0/714784630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00340f56c0 0x7f00340f5ae0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f0024004660 tx=0x7f0024009b30).stop 2023-12-21T03:56:13.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.332+0000 7f0039dfc700 1 -- 172.21.15.78:0/714784630 shutdown_connections 2023-12-21T03:56:13.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.332+0000 7f0039dfc700 1 --2- 172.21.15.78:0/714784630 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00340f6960 0x7f00340f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:13.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.332+0000 7f0039dfc700 1 --2- 172.21.15.78:0/714784630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00340f56c0 0x7f00340f5ae0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:13.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.332+0000 7f0039dfc700 1 -- 172.21.15.78:0/714784630 >> 172.21.15.78:0/714784630 conn(0x7f00340f0e20 msgr2=0x7f00340f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.332+0000 7f0039dfc700 1 -- 172.21.15.78:0/714784630 shutdown_connections 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.332+0000 7f0039dfc700 1 -- 172.21.15.78:0/714784630 wait complete. 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f0039dfc700 1 Processor -- start 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f0039dfc700 1 -- start start 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f0039dfc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00340f56c0 0x7f00340f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f0039dfc700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00340f6960 0x7f00340f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f0039dfc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340f8200 con 0x7f00340f56c0 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f0039dfc700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340f8340 con 0x7f00340f6960 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f00337fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00340f56c0 0x7f00340f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f00337fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00340f56c0 0x7f00340f9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44862/0 (socket says 172.21.15.78:44862) 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f00337fe700 1 -- 172.21.15.78:0/2514621585 learned_addr learned my addr 172.21.15.78:0/2514621585 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f00337fe700 1 -- 172.21.15.78:0/2514621585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00340f6960 msgr2=0x7f00340f7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f00337fe700 1 --2- 172.21.15.78:0/2514621585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00340f6960 0x7f00340f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:56:13.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f00337fe700 1 -- 172.21.15.78:0/2514621585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0024005040 con 0x7f00340f56c0 2023-12-21T03:56:13.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.333+0000 7f00337fe700 1 --2- 172.21.15.78:0/2514621585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00340f56c0 0x7f00340f9820 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f0024009f50 tx=0x7f0024009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:13.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.334+0000 7f002bfff700 1 -- 172.21.15.78:0/2514621585 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0024018070 con 0x7f00340f56c0 2023-12-21T03:56:13.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.334+0000 7f002bfff700 1 -- 172.21.15.78:0/2514621585 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0024013410 con 0x7f00340f56c0 2023-12-21T03:56:13.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.334+0000 7f0039dfc700 1 -- 172.21.15.78:0/2514621585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00340f85c0 con 0x7f00340f56c0 2023-12-21T03:56:13.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.334+0000 7f002bfff700 1 -- 172.21.15.78:0/2514621585 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f002401c620 con 0x7f00340f56c0 2023-12-21T03:56:13.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.334+0000 7f0039dfc700 1 -- 172.21.15.78:0/2514621585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00340f8b20 con 0x7f00340f56c0 2023-12-21T03:56:13.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.335+0000 7f0039dfc700 1 -- 172.21.15.78:0/2514621585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0034045ad0 con 0x7f00340f56c0 2023-12-21T03:56:13.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.335+0000 7f002bfff700 1 -- 172.21.15.78:0/2514621585 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f002401d420 con 0x7f00340f56c0 2023-12-21T03:56:13.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.335+0000 7f002bfff700 1 --2- 172.21.15.78:0/2514621585 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f002005c2a0 0x7f002005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:13.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.335+0000 7f002bfff700 1 -- 172.21.15.78:0/2514621585 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0024077950 con 0x7f00340f56c0 2023-12-21T03:56:13.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.335+0000 7f0032ffd700 1 --2- 172.21.15.78:0/2514621585 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f002005c2a0 0x7f002005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:13.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.336+0000 7f0032ffd700 1 --2- 172.21.15.78:0/2514621585 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f002005c2a0 0x7f002005e760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f001c004880 tx=0x7f001c007ae0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:13.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.339+0000 7f002bfff700 1 -- 172.21.15.78:0/2514621585 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f002404a9d0 con 0x7f00340f56c0 2023-12-21T03:56:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.546+0000 7f0039dfc700 1 -- 172.21.15.78:0/2514621585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f003410f7b0 con 0x7f00340f56c0 2023-12-21T03:56:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.547+0000 7f002bfff700 1 -- 172.21.15.78:0/2514621585 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f002404dff0 con 0x7f00340f56c0 2023-12-21T03:56:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.549+0000 7f0039dfc700 1 -- 172.21.15.78:0/2514621585 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f002005c2a0 msgr2=0x7f002005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.549+0000 7f0039dfc700 1 --2- 172.21.15.78:0/2514621585 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f002005c2a0 0x7f002005e760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f001c004880 tx=0x7f001c007ae0).stop 2023-12-21T03:56:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.549+0000 7f0039dfc700 1 -- 172.21.15.78:0/2514621585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00340f56c0 msgr2=0x7f00340f9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.549+0000 7f0039dfc700 1 --2- 172.21.15.78:0/2514621585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00340f56c0 0x7f00340f9820 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f0024009f50 tx=0x7f0024009b30).stop 2023-12-21T03:56:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.549+0000 7f0039dfc700 1 -- 172.21.15.78:0/2514621585 shutdown_connections 2023-12-21T03:56:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.549+0000 7f0039dfc700 1 --2- 172.21.15.78:0/2514621585 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f002005c2a0 0x7f002005e760 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.549+0000 7f0039dfc700 1 --2- 172.21.15.78:0/2514621585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00340f56c0 0x7f00340f9820 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.549+0000 7f0039dfc700 1 --2- 172.21.15.78:0/2514621585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00340f6960 0x7f00340f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:56:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.549+0000 7f0039dfc700 1 -- 172.21.15.78:0/2514621585 >> 172.21.15.78:0/2514621585 conn(0x7f00340f0e20 msgr2=0x7f00340603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.549+0000 7f0039dfc700 1 -- 172.21.15.78:0/2514621585 shutdown_connections 2023-12-21T03:56:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:13.549+0000 7f0039dfc700 1 -- 172.21.15.78:0/2514621585 wait complete. 2023-12-21T03:56:13.908 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:11.685321+0000 mgr.smithi078.zidbgq (mgr.14162) 565 : cluster [DBG] pgmap v526: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:13.908 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:13.548267+0000 mon.smithi078 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.78:0/2514621585' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:13.950 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:14.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:11.685321+0000 mgr.smithi078.zidbgq (mgr.14162) 565 : cluster [DBG] pgmap v526: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:14.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:13.548267+0000 mon.smithi078 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.78:0/2514621585' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:16.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:13.685626+0000 mgr.smithi078.zidbgq (mgr.14162) 566 : cluster [DBG] pgmap v527: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:16.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:13.685626+0000 mgr.smithi078.zidbgq (mgr.14162) 566 : cluster [DBG] pgmap v527: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:16.951 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:17.189 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:18.006 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:15.686001+0000 mgr.smithi078.zidbgq (mgr.14162) 567 : cluster [DBG] pgmap v528: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:18.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:15.686001+0000 mgr.smithi078.zidbgq (mgr.14162) 567 : cluster [DBG] pgmap v528: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.173+0000 7f25adde7700 1 -- 172.21.15.78:0/1683765211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80f6d60 msgr2=0x7f25a80f7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.173+0000 7f25adde7700 1 --2- 172.21.15.78:0/1683765211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80f6d60 0x7f25a80f7180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2598004660 tx=0x7f2598009b30).stop 2023-12-21T03:56:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.173+0000 7f25adde7700 1 -- 172.21.15.78:0/1683765211 shutdown_connections 2023-12-21T03:56:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.173+0000 7f25adde7700 1 --2- 172.21.15.78:0/1683765211 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a80f5a70 0x7f25a80f5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:56:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.173+0000 7f25adde7700 1 --2- 172.21.15.78:0/1683765211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80f6d60 0x7f25a80f7180 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.173+0000 7f25adde7700 1 -- 172.21.15.78:0/1683765211 >> 172.21.15.78:0/1683765211 conn(0x7f25a80f05e0 msgr2=0x7f25a80f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.173+0000 7f25adde7700 1 -- 172.21.15.78:0/1683765211 shutdown_connections 2023-12-21T03:56:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.173+0000 7f25adde7700 1 -- 172.21.15.78:0/1683765211 wait complete. 2023-12-21T03:56:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.173+0000 7f25adde7700 1 Processor -- start 2023-12-21T03:56:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.173+0000 7f25adde7700 1 -- start start 2023-12-21T03:56:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25adde7700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a80f5a70 0x7f25a80f9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25adde7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80f6d60 0x7f25a80f7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25adde7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a80f83b0 con 0x7f25a80f6d60 2023-12-21T03:56:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25adde7700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a80f84f0 con 0x7f25a80f5a70 2023-12-21T03:56:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25a6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80f6d60 0x7f25a80f7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25a6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80f6d60 0x7f25a80f7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44888/0 (socket says 172.21.15.78:44888) 2023-12-21T03:56:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25a6ffd700 1 -- 172.21.15.78:0/2265055359 learned_addr learned my addr 172.21.15.78:0/2265055359 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25a77fe700 1 --2- 172.21.15.78:0/2265055359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a80f5a70 0x7f25a80f9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25a6ffd700 1 -- 172.21.15.78:0/2265055359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a80f5a70 msgr2=0x7f25a80f9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25a6ffd700 1 --2- 172.21.15.78:0/2265055359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a80f5a70 0x7f25a80f9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25a6ffd700 1 -- 172.21.15.78:0/2265055359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598005040 con 0x7f25a80f6d60 2023-12-21T03:56:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25a6ffd700 1 --2- 172.21.15.78:0/2265055359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80f6d60 0x7f25a80f7dc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f259c00b410 tx=0x7f259c006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f258ffff700 1 -- 172.21.15.78:0/2265055359 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f259c0125f0 con 0x7f25a80f6d60 2023-12-21T03:56:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25adde7700 1 -- 172.21.15.78:0/2265055359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a80f87d0 con 0x7f25a80f6d60 2023-12-21T03:56:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f258ffff700 1 -- 172.21.15.78:0/2265055359 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f259c012c00 con 0x7f25a80f6d60 2023-12-21T03:56:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f258ffff700 1 -- 172.21.15.78:0/2265055359 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f259c002aa0 con 0x7f25a80f6d60 2023-12-21T03:56:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.174+0000 7f25adde7700 1 -- 172.21.15.78:0/2265055359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a810f100 con 0x7f25a80f6d60 2023-12-21T03:56:18.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.175+0000 7f25adde7700 1 -- 172.21.15.78:0/2265055359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a8045ad0 con 0x7f25a80f6d60 2023-12-21T03:56:18.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.176+0000 7f258ffff700 1 -- 172.21.15.78:0/2265055359 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f259c00f030 con 0x7f25a80f6d60 2023-12-21T03:56:18.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.176+0000 7f258ffff700 1 --2- 172.21.15.78:0/2265055359 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f259005c2f0 0x7f259005e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:18.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.176+0000 7f258ffff700 1 -- 172.21.15.78:0/2265055359 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f259c076100 con 0x7f25a80f6d60 2023-12-21T03:56:18.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.176+0000 7f25a77fe700 1 --2- 172.21.15.78:0/2265055359 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f259005c2f0 0x7f259005e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:18.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.176+0000 7f25a77fe700 1 --2- 172.21.15.78:0/2265055359 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f259005c2f0 0x7f259005e7b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f2598009d60 tx=0x7f2598009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.180+0000 7f258ffff700 1 -- 172.21.15.78:0/2265055359 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f259c048fe0 con 0x7f25a80f6d60 2023-12-21T03:56:18.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.388+0000 7f25adde7700 1 -- 172.21.15.78:0/2265055359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f25a810f7d0 con 0x7f25a80f6d60 2023-12-21T03:56:18.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.388+0000 7f258ffff700 1 -- 172.21.15.78:0/2265055359 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f259c04c600 con 0x7f25a80f6d60 2023-12-21T03:56:18.390 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:18.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.390+0000 7f25adde7700 1 -- 172.21.15.78:0/2265055359 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f259005c2f0 msgr2=0x7f259005e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:18.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.390+0000 7f25adde7700 1 --2- 172.21.15.78:0/2265055359 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f259005c2f0 0x7f259005e7b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f2598009d60 tx=0x7f2598009b30).stop 2023-12-21T03:56:18.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.390+0000 7f25adde7700 1 -- 172.21.15.78:0/2265055359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80f6d60 msgr2=0x7f25a80f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:18.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.390+0000 7f25adde7700 1 --2- 172.21.15.78:0/2265055359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80f6d60 0x7f25a80f7dc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f259c00b410 tx=0x7f259c006ef0).stop 2023-12-21T03:56:18.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.390+0000 7f25adde7700 1 -- 172.21.15.78:0/2265055359 shutdown_connections 2023-12-21T03:56:18.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.390+0000 7f25adde7700 1 --2- 172.21.15.78:0/2265055359 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f259005c2f0 0x7f259005e7b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:18.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.390+0000 7f25adde7700 1 --2- 172.21.15.78:0/2265055359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a80f5a70 0x7f25a80f9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:18.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.390+0000 7f25adde7700 1 --2- 172.21.15.78:0/2265055359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80f6d60 0x7f25a80f7dc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:18.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.390+0000 7f25adde7700 1 -- 172.21.15.78:0/2265055359 >> 172.21.15.78:0/2265055359 conn(0x7f25a80f05e0 msgr2=0x7f25a80f5140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:18.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.390+0000 7f25adde7700 1 -- 172.21.15.78:0/2265055359 shutdown_connections 2023-12-21T03:56:18.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:18.390+0000 7f25adde7700 1 -- 172.21.15.78:0/2265055359 wait complete. 2023-12-21T03:56:18.707 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:18 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:18.389800+0000 mon.smithi078 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.78:0/2265055359' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:18.766 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:19.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:18 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:18.389800+0000 mon.smithi078 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.78:0/2265055359' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:20.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:17.686335+0000 mgr.smithi078.zidbgq (mgr.14162) 568 : cluster [DBG] pgmap v529: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:20.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:17.686335+0000 mgr.smithi078.zidbgq (mgr.14162) 568 : cluster [DBG] pgmap v529: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:21.767 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:22.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:19.686831+0000 mgr.smithi078.zidbgq (mgr.14162) 569 : cluster [DBG] pgmap v530: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:22.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:19.686831+0000 mgr.smithi078.zidbgq (mgr.14162) 569 : cluster [DBG] pgmap v530: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:23.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.055+0000 7f7f2b344700 1 -- 172.21.15.78:0/2110741277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f240f7a60 msgr2=0x7f7f240f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:23.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.055+0000 7f7f2b344700 1 --2- 172.21.15.78:0/2110741277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f240f7a60 0x7f7f240f7ec0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f7f20004660 tx=0x7f7f20009b30).stop 2023-12-21T03:56:23.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.055+0000 7f7f2b344700 1 -- 172.21.15.78:0/2110741277 shutdown_connections 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.055+0000 7f7f2b344700 1 --2- 172.21.15.78:0/2110741277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f240f7a60 0x7f7f240f7ec0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.055+0000 7f7f2b344700 1 --2- 172.21.15.78:0/2110741277 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f240f67c0 0x7f7f240f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.055+0000 7f7f2b344700 1 -- 172.21.15.78:0/2110741277 >> 172.21.15.78:0/2110741277 conn(0x7f7f240f1f20 msgr2=0x7f7f240f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.055+0000 7f7f2b344700 1 -- 172.21.15.78:0/2110741277 shutdown_connections 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f2b344700 1 -- 172.21.15.78:0/2110741277 wait complete. 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f2b344700 1 Processor -- start 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f2b344700 1 -- start start 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f2b344700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f240f67c0 0x7f7f241070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f2b344700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f240f7a60 0x7f7f241075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f2b344700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f24107b50 con 0x7f7f240f67c0 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f2b344700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f24107c90 con 0x7f7f240f7a60 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f290e0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f240f67c0 0x7f7f241070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f290e0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f240f67c0 0x7f7f241070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45156/0 (socket says 172.21.15.78:45156) 2023-12-21T03:56:23.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f290e0700 1 -- 172.21.15.78:0/358050461 learned_addr learned my addr 172.21.15.78:0/358050461 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:23.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f288df700 1 --2- 172.21.15.78:0/358050461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f240f7a60 0x7f7f241075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:23.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f290e0700 1 -- 172.21.15.78:0/358050461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f240f7a60 msgr2=0x7f7f241075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:23.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.056+0000 7f7f290e0700 1 --2- 172.21.15.78:0/358050461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f240f7a60 0x7f7f241075a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:23.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.057+0000 7f7f290e0700 1 -- 172.21.15.78:0/358050461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f20005040 con 0x7f7f240f67c0 2023-12-21T03:56:23.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.057+0000 7f7f290e0700 1 --2- 172.21.15.78:0/358050461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f240f67c0 0x7f7f241070b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f7f14002700 tx=0x7f7f14009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:23.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.057+0000 7f7f19ffb700 1 -- 172.21.15.78:0/358050461 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f14009320 con 0x7f7f240f67c0 2023-12-21T03:56:23.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.057+0000 7f7f2b344700 1 -- 172.21.15.78:0/358050461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f24108630 con 0x7f7f240f67c0 2023-12-21T03:56:23.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.057+0000 7f7f19ffb700 1 -- 172.21.15.78:0/358050461 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f1400f040 con 0x7f7f240f67c0 2023-12-21T03:56:23.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.057+0000 7f7f19ffb700 1 -- 172.21.15.78:0/358050461 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f14014900 con 0x7f7f240f67c0 2023-12-21T03:56:23.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.057+0000 7f7f2b344700 1 -- 172.21.15.78:0/358050461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f24108bf0 con 0x7f7f240f67c0 2023-12-21T03:56:23.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.058+0000 7f7f2b344700 1 -- 172.21.15.78:0/358050461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f24045ad0 con 0x7f7f240f67c0 2023-12-21T03:56:23.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.058+0000 7f7f19ffb700 1 -- 172.21.15.78:0/358050461 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7f14014a60 con 0x7f7f240f67c0 2023-12-21T03:56:23.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.058+0000 7f7f19ffb700 1 --2- 172.21.15.78:0/358050461 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7f1005c2a0 0x7f7f1005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:23.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.058+0000 7f7f19ffb700 1 -- 172.21.15.78:0/358050461 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7f14075e40 con 0x7f7f240f67c0 2023-12-21T03:56:23.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.058+0000 7f7f288df700 1 --2- 172.21.15.78:0/358050461 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7f1005c2a0 0x7f7f1005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:23.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.059+0000 7f7f288df700 1 --2- 172.21.15.78:0/358050461 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7f1005c2a0 0x7f7f1005e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f7f20004e30 tx=0x7f7f20009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:23.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.062+0000 7f7f19ffb700 1 -- 172.21.15.78:0/358050461 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f14048e40 con 0x7f7f240f67c0 2023-12-21T03:56:23.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.267+0000 7f7f2b344700 1 -- 172.21.15.78:0/358050461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7f240ff0d0 con 0x7f7f240f67c0 2023-12-21T03:56:23.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.267+0000 7f7f19ffb700 1 -- 172.21.15.78:0/358050461 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f7f1404c460 con 0x7f7f240f67c0 2023-12-21T03:56:23.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:23.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.269+0000 7f7f2b344700 1 -- 172.21.15.78:0/358050461 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7f1005c2a0 msgr2=0x7f7f1005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:23.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.269+0000 7f7f2b344700 1 --2- 172.21.15.78:0/358050461 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7f1005c2a0 0x7f7f1005e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f7f20004e30 tx=0x7f7f20009b30).stop 2023-12-21T03:56:23.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.269+0000 7f7f2b344700 1 -- 172.21.15.78:0/358050461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f240f67c0 msgr2=0x7f7f241070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:23.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.269+0000 7f7f2b344700 1 --2- 172.21.15.78:0/358050461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f240f67c0 0x7f7f241070b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f7f14002700 tx=0x7f7f14009d80).stop 2023-12-21T03:56:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.269+0000 7f7f2b344700 1 -- 172.21.15.78:0/358050461 shutdown_connections 2023-12-21T03:56:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.269+0000 7f7f2b344700 1 --2- 172.21.15.78:0/358050461 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f7f1005c2a0 0x7f7f1005e760 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.269+0000 7f7f2b344700 1 --2- 172.21.15.78:0/358050461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f240f67c0 0x7f7f241070b0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.269+0000 7f7f2b344700 1 --2- 172.21.15.78:0/358050461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f240f7a60 0x7f7f241075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.269+0000 7f7f2b344700 1 -- 172.21.15.78:0/358050461 >> 172.21.15.78:0/358050461 conn(0x7f7f240f1f20 msgr2=0x7f7f240fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.269+0000 7f7f2b344700 1 -- 172.21.15.78:0/358050461 shutdown_connections 2023-12-21T03:56:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:23.269+0000 7f7f2b344700 1 -- 172.21.15.78:0/358050461 wait complete. 2023-12-21T03:56:23.666 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:24.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:21.687174+0000 mgr.smithi078.zidbgq (mgr.14162) 570 : cluster [DBG] pgmap v531: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:24.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:23.268692+0000 mon.smithi078 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.78:0/358050461' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:24.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:21.687174+0000 mgr.smithi078.zidbgq (mgr.14162) 570 : cluster [DBG] pgmap v531: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:24.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:23.268692+0000 mon.smithi078 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.78:0/358050461' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:26.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:23.687555+0000 mgr.smithi078.zidbgq (mgr.14162) 571 : cluster [DBG] pgmap v532: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:26.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:23.687555+0000 mgr.smithi078.zidbgq (mgr.14162) 571 : cluster [DBG] pgmap v532: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:26.669 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:27.696 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:25.688036+0000 mgr.smithi078.zidbgq (mgr.14162) 572 : cluster [DBG] pgmap v533: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:27.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.892+0000 7fb382d6e700 1 -- 172.21.15.78:0/3920322046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb37c0f48e0 msgr2=0x7fb37c0f4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:27.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.892+0000 7fb382d6e700 1 --2- 172.21.15.78:0/3920322046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb37c0f48e0 0x7fb37c0f4d00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fb378004660 tx=0x7fb378009b30).stop 2023-12-21T03:56:27.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.892+0000 7fb382d6e700 1 -- 172.21.15.78:0/3920322046 shutdown_connections 2023-12-21T03:56:27.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.892+0000 7fb382d6e700 1 --2- 172.21.15.78:0/3920322046 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb37c0f5280 0x7fb37c0f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:27.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.892+0000 7fb382d6e700 1 --2- 172.21.15.78:0/3920322046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb37c0f48e0 0x7fb37c0f4d00 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:27.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.892+0000 7fb382d6e700 1 -- 172.21.15.78:0/3920322046 >> 172.21.15.78:0/3920322046 conn(0x7fb37c0f0400 msgr2=0x7fb37c0f2820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:27.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.893+0000 7fb382d6e700 1 -- 172.21.15.78:0/3920322046 shutdown_connections 2023-12-21T03:56:27.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.893+0000 7fb382d6e700 1 -- 172.21.15.78:0/3920322046 wait complete. 2023-12-21T03:56:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.893+0000 7fb382d6e700 1 Processor -- start 2023-12-21T03:56:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.893+0000 7fb382d6e700 1 -- start start 2023-12-21T03:56:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.893+0000 7fb382d6e700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb37c0f48e0 0x7fb37c106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.893+0000 7fb382d6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb37c0f5280 0x7fb37c1073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.893+0000 7fb382d6e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c107980 con 0x7fb37c0f5280 2023-12-21T03:56:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.893+0000 7fb382d6e700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c107ac0 con 0x7fb37c0f48e0 2023-12-21T03:56:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb38156b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb37c0f5280 0x7fb37c1073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb38156b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb37c0f5280 0x7fb37c1073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45172/0 (socket says 172.21.15.78:45172) 2023-12-21T03:56:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb38156b700 1 -- 172.21.15.78:0/3454996274 learned_addr learned my addr 172.21.15.78:0/3454996274 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb381d6c700 1 --2- 172.21.15.78:0/3454996274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb37c0f48e0 0x7fb37c106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb38156b700 1 -- 172.21.15.78:0/3454996274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb37c0f48e0 msgr2=0x7fb37c106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:27.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb38156b700 1 --2- 172.21.15.78:0/3454996274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb37c0f48e0 0x7fb37c106ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:27.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb38156b700 1 -- 172.21.15.78:0/3454996274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb378005040 con 0x7fb37c0f5280 2023-12-21T03:56:27.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb38156b700 1 --2- 172.21.15.78:0/3454996274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb37c0f5280 0x7fb37c1073d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7fb36c0026d0 tx=0x7fb36c009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:27.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb3727fc700 1 -- 172.21.15.78:0/3454996274 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36c0079c0 con 0x7fb37c0f5280 2023-12-21T03:56:27.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb3727fc700 1 -- 172.21.15.78:0/3454996274 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb36c00f040 con 0x7fb37c0f5280 2023-12-21T03:56:27.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb3727fc700 1 -- 172.21.15.78:0/3454996274 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36c014850 con 0x7fb37c0f5280 2023-12-21T03:56:27.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb382d6e700 1 -- 172.21.15.78:0/3454996274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37c108460 con 0x7fb37c0f5280 2023-12-21T03:56:27.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.894+0000 7fb382d6e700 1 -- 172.21.15.78:0/3454996274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37c108a20 con 0x7fb37c0f5280 2023-12-21T03:56:27.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.895+0000 7fb382d6e700 1 -- 172.21.15.78:0/3454996274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37c045ad0 con 0x7fb37c0f5280 2023-12-21T03:56:27.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.895+0000 7fb3727fc700 1 -- 172.21.15.78:0/3454996274 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb36c007b20 con 0x7fb37c0f5280 2023-12-21T03:56:27.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.896+0000 7fb3727fc700 1 --2- 172.21.15.78:0/3454996274 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb36805c2b0 0x7fb36805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:27.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.896+0000 7fb3727fc700 1 -- 172.21.15.78:0/3454996274 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb36c075a60 con 0x7fb37c0f5280 2023-12-21T03:56:27.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.896+0000 7fb381d6c700 1 --2- 172.21.15.78:0/3454996274 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb36805c2b0 0x7fb36805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:27.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.896+0000 7fb381d6c700 1 --2- 172.21.15.78:0/3454996274 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb36805c2b0 0x7fb36805e770 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fb378009f50 tx=0x7fb378009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:27.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:27.900+0000 7fb3727fc700 1 -- 172.21.15.78:0/3454996274 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb36c048ae0 con 0x7fb37c0f5280 2023-12-21T03:56:28.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:25.688036+0000 mgr.smithi078.zidbgq (mgr.14162) 572 : cluster [DBG] pgmap v533: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:28.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.109+0000 7fb382d6e700 1 -- 172.21.15.78:0/3454996274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb37c027a00 con 0x7fb37c0f5280 2023-12-21T03:56:28.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.109+0000 7fb3727fc700 1 -- 172.21.15.78:0/3454996274 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fb36c04c100 con 0x7fb37c0f5280 2023-12-21T03:56:28.110 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:28.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.111+0000 7fb382d6e700 1 -- 172.21.15.78:0/3454996274 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb36805c2b0 msgr2=0x7fb36805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:28.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.111+0000 7fb382d6e700 1 --2- 172.21.15.78:0/3454996274 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb36805c2b0 0x7fb36805e770 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fb378009f50 tx=0x7fb378009b30).stop 2023-12-21T03:56:28.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.111+0000 7fb382d6e700 1 -- 172.21.15.78:0/3454996274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb37c0f5280 msgr2=0x7fb37c1073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:28.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.111+0000 7fb382d6e700 1 --2- 172.21.15.78:0/3454996274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb37c0f5280 0x7fb37c1073d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7fb36c0026d0 tx=0x7fb36c009e00).stop 2023-12-21T03:56:28.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.111+0000 7fb382d6e700 1 -- 172.21.15.78:0/3454996274 shutdown_connections 2023-12-21T03:56:28.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.111+0000 7fb382d6e700 1 --2- 172.21.15.78:0/3454996274 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fb36805c2b0 0x7fb36805e770 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:28.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.111+0000 7fb382d6e700 1 --2- 172.21.15.78:0/3454996274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb37c0f48e0 0x7fb37c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:28.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.111+0000 7fb382d6e700 1 --2- 172.21.15.78:0/3454996274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb37c0f5280 0x7fb37c1073d0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:28.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.111+0000 7fb382d6e700 1 -- 172.21.15.78:0/3454996274 >> 172.21.15.78:0/3454996274 conn(0x7fb37c0f0400 msgr2=0x7fb37c0fb450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:28.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.111+0000 7fb382d6e700 1 -- 172.21.15.78:0/3454996274 shutdown_connections 2023-12-21T03:56:28.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:28.111+0000 7fb382d6e700 1 -- 172.21.15.78:0/3454996274 wait complete. 2023-12-21T03:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:28.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:28 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:28.109669+0000 mon.smithi078 (mon.0) 577 : audit [DBG] from='client.? 172.21.15.78:0/3454996274' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:29.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:28 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:28.109669+0000 mon.smithi078 (mon.0) 577 : audit [DBG] from='client.? 172.21.15.78:0/3454996274' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:30.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:27.688360+0000 mgr.smithi078.zidbgq (mgr.14162) 573 : cluster [DBG] pgmap v534: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:30.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:27.688360+0000 mgr.smithi078.zidbgq (mgr.14162) 573 : cluster [DBG] pgmap v534: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:31.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:29.688732+0000 mgr.smithi078.zidbgq (mgr.14162) 574 : cluster [DBG] pgmap v535: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:31.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:30.573434+0000 mon.smithi078 (mon.0) 578 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:56:31.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:30 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:30.573664+0000 mon.smithi078 (mon.0) 579 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:56:31.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:29.688732+0000 mgr.smithi078.zidbgq (mgr.14162) 574 : cluster [DBG] pgmap v535: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:31.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:30.573434+0000 mon.smithi078 (mon.0) 578 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:56:31.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:30 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:30.573664+0000 mon.smithi078 (mon.0) 579 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:56:31.443 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:31.678 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:32.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.589+0000 7f0b9ffff700 1 -- 172.21.15.78:0/639111131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba00f62f0 msgr2=0x7f0ba00f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:32.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.589+0000 7f0b9ffff700 1 --2- 172.21.15.78:0/639111131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba00f62f0 0x7f0ba00f6750 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f0b94004660 tx=0x7f0b94009b30).stop 2023-12-21T03:56:32.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.589+0000 7f0b9ffff700 1 -- 172.21.15.78:0/639111131 shutdown_connections 2023-12-21T03:56:32.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.589+0000 7f0b9ffff700 1 --2- 172.21.15.78:0/639111131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba00f62f0 0x7f0ba00f6750 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.589+0000 7f0b9ffff700 1 --2- 172.21.15.78:0/639111131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba00f7ce0 0x7f0ba00f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.589+0000 7f0b9ffff700 1 -- 172.21.15.78:0/639111131 >> 172.21.15.78:0/639111131 conn(0x7f0ba00f1d70 msgr2=0x7f0ba00f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.589+0000 7f0b9ffff700 1 -- 172.21.15.78:0/639111131 shutdown_connections 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.589+0000 7f0b9ffff700 1 -- 172.21.15.78:0/639111131 wait complete. 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.590+0000 7f0b9ffff700 1 Processor -- start 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.590+0000 7f0b9ffff700 1 -- start start 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba00f62f0 0x7f0ba0106fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9ffff700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba00f7ce0 0x7f0ba0107490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9ffff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba0107a40 con 0x7f0ba00f62f0 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9ffff700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba0107b80 con 0x7f0ba00f7ce0 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba00f62f0 0x7f0ba0106fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba00f62f0 0x7f0ba0106fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39250/0 (socket says 172.21.15.78:39250) 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9effd700 1 -- 172.21.15.78:0/3423160823 learned_addr learned my addr 172.21.15.78:0/3423160823 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9e7fc700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba00f7ce0 0x7f0ba0107490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9effd700 1 -- 172.21.15.78:0/3423160823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba00f7ce0 msgr2=0x7f0ba0107490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9effd700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba00f7ce0 0x7f0ba0107490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9effd700 1 -- 172.21.15.78:0/3423160823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b94005040 con 0x7f0ba00f62f0 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9e7fc700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba00f7ce0 0x7f0ba0107490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9effd700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba00f62f0 0x7f0ba0106fa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f0b9000c530 tx=0x7f0b90007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b877fe700 1 -- 172.21.15.78:0/3423160823 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b9000f650 con 0x7f0ba00f62f0 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9ffff700 1 -- 172.21.15.78:0/3423160823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba0108520 con 0x7f0ba00f62f0 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b877fe700 1 -- 172.21.15.78:0/3423160823 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b9000fc60 con 0x7f0ba00f62f0 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b877fe700 1 -- 172.21.15.78:0/3423160823 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b90012d20 con 0x7f0ba00f62f0 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.591+0000 7f0b9ffff700 1 -- 172.21.15.78:0/3423160823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba0108ae0 con 0x7f0ba00f62f0 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.592+0000 7f0b9ffff700 1 -- 172.21.15.78:0/3423160823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba0045ad0 con 0x7f0ba00f62f0 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.593+0000 7f0b877fe700 1 -- 172.21.15.78:0/3423160823 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0b90019030 con 0x7f0ba00f62f0 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.593+0000 7f0b877fe700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0b8805c2b0 0x7f0b8805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.593+0000 7f0b877fe700 1 -- 172.21.15.78:0/3423160823 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0b900757f0 con 0x7f0ba00f62f0 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.593+0000 7f0b9e7fc700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0b8805c2b0 0x7f0b8805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.593+0000 7f0b9e7fc700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0b8805c2b0 0x7f0b8805e770 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f0b94013580 tx=0x7f0b94009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:32.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.597+0000 7f0b877fe700 1 -- 172.21.15.78:0/3423160823 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b90048840 con 0x7f0ba00f62f0 2023-12-21T03:56:32.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.806+0000 7f0b9ffff700 1 -- 172.21.15.78:0/3423160823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0ba00008d0 con 0x7f0ba00f62f0 2023-12-21T03:56:32.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.806+0000 7f0b877fe700 1 -- 172.21.15.78:0/3423160823 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f0b90017080 con 0x7f0ba00f62f0 2023-12-21T03:56:32.808 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:32.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.808+0000 7f0b9ffff700 1 -- 172.21.15.78:0/3423160823 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0b8805c2b0 msgr2=0x7f0b8805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:32.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.808+0000 7f0b9ffff700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0b8805c2b0 0x7f0b8805e770 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f0b94013580 tx=0x7f0b94009b30).stop 2023-12-21T03:56:32.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.808+0000 7f0b9ffff700 1 -- 172.21.15.78:0/3423160823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba00f62f0 msgr2=0x7f0ba0106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:32.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.808+0000 7f0b9ffff700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba00f62f0 0x7f0ba0106fa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f0b9000c530 tx=0x7f0b90007e10).stop 2023-12-21T03:56:32.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.808+0000 7f0b9ffff700 1 -- 172.21.15.78:0/3423160823 shutdown_connections 2023-12-21T03:56:32.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.808+0000 7f0b9ffff700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f0b8805c2b0 0x7f0b8805e770 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:32.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.808+0000 7f0b9ffff700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba00f62f0 0x7f0ba0106fa0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:32.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.808+0000 7f0b9ffff700 1 --2- 172.21.15.78:0/3423160823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba00f7ce0 0x7f0ba0107490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:32.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.808+0000 7f0b9ffff700 1 -- 172.21.15.78:0/3423160823 >> 172.21.15.78:0/3423160823 conn(0x7f0ba00f1d70 msgr2=0x7f0ba00fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:32.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.808+0000 7f0b9ffff700 1 -- 172.21.15.78:0/3423160823 shutdown_connections 2023-12-21T03:56:32.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:32.808+0000 7f0b9ffff700 1 -- 172.21.15.78:0/3423160823 wait complete. 2023-12-21T03:56:33.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:32 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:31.689114+0000 mgr.smithi078.zidbgq (mgr.14162) 575 : cluster [DBG] pgmap v536: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:33.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:32 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:31.689114+0000 mgr.smithi078.zidbgq (mgr.14162) 575 : cluster [DBG] pgmap v536: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:33.190 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:34.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:32.807936+0000 mon.smithi078 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.78:0/3423160823' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:34.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:32.807936+0000 mon.smithi078 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.78:0/3423160823' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:35.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:33.689455+0000 mgr.smithi078.zidbgq (mgr.14162) 576 : cluster [DBG] pgmap v537: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:35.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:33.689455+0000 mgr.smithi078.zidbgq (mgr.14162) 576 : cluster [DBG] pgmap v537: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:36.190 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:36.427 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.432+0000 7f30d7c7b700 1 -- 172.21.15.78:0/1237214922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30d00f7a60 msgr2=0x7f30d00f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.432+0000 7f30d7c7b700 1 --2- 172.21.15.78:0/1237214922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30d00f7a60 0x7f30d00f7ec0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f30cc004660 tx=0x7f30cc009b30).stop 2023-12-21T03:56:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.432+0000 7f30d7c7b700 1 -- 172.21.15.78:0/1237214922 shutdown_connections 2023-12-21T03:56:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.432+0000 7f30d7c7b700 1 --2- 172.21.15.78:0/1237214922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30d00f7a60 0x7f30d00f7ec0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.432+0000 7f30d7c7b700 1 --2- 172.21.15.78:0/1237214922 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30d00f67c0 0x7f30d00f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.432+0000 7f30d7c7b700 1 -- 172.21.15.78:0/1237214922 >> 172.21.15.78:0/1237214922 conn(0x7f30d00f1f20 msgr2=0x7f30d00f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.432+0000 7f30d7c7b700 1 -- 172.21.15.78:0/1237214922 shutdown_connections 2023-12-21T03:56:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.432+0000 7f30d7c7b700 1 -- 172.21.15.78:0/1237214922 wait complete. 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d7c7b700 1 Processor -- start 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d7c7b700 1 -- start start 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d7c7b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30d00f67c0 0x7f30d006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d7c7b700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30d00f7a60 0x7f30d0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d7c7b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d006a380 con 0x7f30d00f67c0 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d7c7b700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d006a4c0 con 0x7f30d00f7a60 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d5a17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30d00f67c0 0x7f30d006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d5a17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30d00f67c0 0x7f30d006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39254/0 (socket says 172.21.15.78:39254) 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d5a17700 1 -- 172.21.15.78:0/2638572306 learned_addr learned my addr 172.21.15.78:0/2638572306 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d5216700 1 --2- 172.21.15.78:0/2638572306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30d00f7a60 0x7f30d0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d5a17700 1 -- 172.21.15.78:0/2638572306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30d00f7a60 msgr2=0x7f30d0069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d5a17700 1 --2- 172.21.15.78:0/2638572306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30d00f7a60 0x7f30d0069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d5a17700 1 -- 172.21.15.78:0/2638572306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc005040 con 0x7f30d00f67c0 2023-12-21T03:56:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.433+0000 7f30d5a17700 1 --2- 172.21.15.78:0/2638572306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30d00f67c0 0x7f30d006b9a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f30c0002700 tx=0x7f30c0009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.434+0000 7f30c67fc700 1 -- 172.21.15.78:0/2638572306 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30c00079c0 con 0x7f30d00f67c0 2023-12-21T03:56:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.434+0000 7f30c67fc700 1 -- 172.21.15.78:0/2638572306 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30c000f040 con 0x7f30d00f67c0 2023-12-21T03:56:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.434+0000 7f30d7c7b700 1 -- 172.21.15.78:0/2638572306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d006a7a0 con 0x7f30d00f67c0 2023-12-21T03:56:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.434+0000 7f30d7c7b700 1 -- 172.21.15.78:0/2638572306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d010f0e0 con 0x7f30d00f67c0 2023-12-21T03:56:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.434+0000 7f30c67fc700 1 -- 172.21.15.78:0/2638572306 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30c0014860 con 0x7f30d00f67c0 2023-12-21T03:56:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.434+0000 7f30d7c7b700 1 -- 172.21.15.78:0/2638572306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b8004fa0 con 0x7f30d00f67c0 2023-12-21T03:56:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.435+0000 7f30c67fc700 1 -- 172.21.15.78:0/2638572306 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f30c001b060 con 0x7f30d00f67c0 2023-12-21T03:56:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.436+0000 7f30c67fc700 1 --2- 172.21.15.78:0/2638572306 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30bc05c2a0 0x7f30bc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.436+0000 7f30c67fc700 1 -- 172.21.15.78:0/2638572306 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f30c00759c0 con 0x7f30d00f67c0 2023-12-21T03:56:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.436+0000 7f30d5216700 1 --2- 172.21.15.78:0/2638572306 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30bc05c2a0 0x7f30bc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.436+0000 7f30d5216700 1 --2- 172.21.15.78:0/2638572306 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30bc05c2a0 0x7f30bc05e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f30cc009f50 tx=0x7f30cc009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:37.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.441+0000 7f30c67fc700 1 -- 172.21.15.78:0/2638572306 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30c00489c0 con 0x7f30d00f67c0 2023-12-21T03:56:37.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:35.689806+0000 mgr.smithi078.zidbgq (mgr.14162) 577 : cluster [DBG] pgmap v538: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.661+0000 7f30d7c7b700 1 -- 172.21.15.78:0/2638572306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f30b8005bf0 con 0x7f30d00f67c0 2023-12-21T03:56:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.661+0000 7f30c67fc700 1 -- 172.21.15.78:0/2638572306 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f30c004bfe0 con 0x7f30d00f67c0 2023-12-21T03:56:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.663+0000 7f30d7c7b700 1 -- 172.21.15.78:0/2638572306 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30bc05c2a0 msgr2=0x7f30bc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.663+0000 7f30d7c7b700 1 --2- 172.21.15.78:0/2638572306 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30bc05c2a0 0x7f30bc05e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f30cc009f50 tx=0x7f30cc009b30).stop 2023-12-21T03:56:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.663+0000 7f30d7c7b700 1 -- 172.21.15.78:0/2638572306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30d00f67c0 msgr2=0x7f30d006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.663+0000 7f30d7c7b700 1 --2- 172.21.15.78:0/2638572306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30d00f67c0 0x7f30d006b9a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f30c0002700 tx=0x7f30c0009d80).stop 2023-12-21T03:56:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.663+0000 7f30d7c7b700 1 -- 172.21.15.78:0/2638572306 shutdown_connections 2023-12-21T03:56:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.663+0000 7f30d7c7b700 1 --2- 172.21.15.78:0/2638572306 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f30bc05c2a0 0x7f30bc05e760 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.663+0000 7f30d7c7b700 1 --2- 172.21.15.78:0/2638572306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30d00f67c0 0x7f30d006b9a0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.663+0000 7f30d7c7b700 1 --2- 172.21.15.78:0/2638572306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30d00f7a60 0x7f30d0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.663+0000 7f30d7c7b700 1 -- 172.21.15.78:0/2638572306 >> 172.21.15.78:0/2638572306 conn(0x7f30d00f1f20 msgr2=0x7f30d00fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.663+0000 7f30d7c7b700 1 -- 172.21.15.78:0/2638572306 shutdown_connections 2023-12-21T03:56:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:37.663+0000 7f30d7c7b700 1 -- 172.21.15.78:0/2638572306 wait complete. 2023-12-21T03:56:37.767 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:35.689806+0000 mgr.smithi078.zidbgq (mgr.14162) 577 : cluster [DBG] pgmap v538: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:38.055 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:38.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:38 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:37.662441+0000 mon.smithi078 (mon.0) 581 : audit [DBG] from='client.? 172.21.15.78:0/2638572306' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:38.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:38 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:37.662441+0000 mon.smithi078 (mon.0) 581 : audit [DBG] from='client.? 172.21.15.78:0/2638572306' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:39.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:37.690242+0000 mgr.smithi078.zidbgq (mgr.14162) 578 : cluster [DBG] pgmap v539: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:39.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:37.690242+0000 mgr.smithi078.zidbgq (mgr.14162) 578 : cluster [DBG] pgmap v539: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:41.056 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:41.294 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:41.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:39.690644+0000 mgr.smithi078.zidbgq (mgr.14162) 579 : cluster [DBG] pgmap v540: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:41.768 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:39.690644+0000 mgr.smithi078.zidbgq (mgr.14162) 579 : cluster [DBG] pgmap v540: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.270+0000 7f5c49332700 1 -- 172.21.15.78:0/3685712890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c440f67f0 msgr2=0x7f5c440f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.270+0000 7f5c49332700 1 --2- 172.21.15.78:0/3685712890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c440f67f0 0x7f5c440f6c50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f5c34004660 tx=0x7f5c34009b30).stop 2023-12-21T03:56:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.270+0000 7f5c49332700 1 -- 172.21.15.78:0/3685712890 shutdown_connections 2023-12-21T03:56:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.270+0000 7f5c49332700 1 --2- 172.21.15.78:0/3685712890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c440f67f0 0x7f5c440f6c50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.270+0000 7f5c49332700 1 --2- 172.21.15.78:0/3685712890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c440f5550 0x7f5c440f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:56:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.270+0000 7f5c49332700 1 -- 172.21.15.78:0/3685712890 >> 172.21.15.78:0/3685712890 conn(0x7f5c440f0cb0 msgr2=0x7f5c440f30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.270+0000 7f5c49332700 1 -- 172.21.15.78:0/3685712890 shutdown_connections 2023-12-21T03:56:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.270+0000 7f5c49332700 1 -- 172.21.15.78:0/3685712890 wait complete. 2023-12-21T03:56:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.271+0000 7f5c49332700 1 Processor -- start 2023-12-21T03:56:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.271+0000 7f5c49332700 1 -- start start 2023-12-21T03:56:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.271+0000 7f5c49332700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c440f5550 0x7f5c441070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.271+0000 7f5c49332700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c441075c0 0x7f5c441084b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.271+0000 7f5c49332700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c44107b20 con 0x7f5c441075c0 2023-12-21T03:56:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.271+0000 7f5c49332700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c44107c60 con 0x7f5c440f5550 2023-12-21T03:56:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.271+0000 7f5c427fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c441075c0 0x7f5c441084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.271+0000 7f5c427fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c441075c0 0x7f5c441084b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54954/0 (socket says 172.21.15.78:54954) 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.271+0000 7f5c427fc700 1 -- 172.21.15.78:0/1598620695 learned_addr learned my addr 172.21.15.78:0/1598620695 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.271+0000 7f5c42ffd700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c440f5550 0x7f5c441070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.272+0000 7f5c427fc700 1 -- 172.21.15.78:0/1598620695 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c440f5550 msgr2=0x7f5c441070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.272+0000 7f5c427fc700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c440f5550 0x7f5c441070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.272+0000 7f5c427fc700 1 -- 172.21.15.78:0/1598620695 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c34005040 con 0x7f5c441075c0 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.272+0000 7f5c42ffd700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c440f5550 0x7f5c441070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.272+0000 7f5c427fc700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c441075c0 0x7f5c441084b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f5c34009f50 tx=0x7f5c34009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.272+0000 7f5c2b7fe700 1 -- 172.21.15.78:0/1598620695 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c34019070 con 0x7f5c441075c0 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.272+0000 7f5c2b7fe700 1 -- 172.21.15.78:0/1598620695 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c34013410 con 0x7f5c441075c0 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.272+0000 7f5c49332700 1 -- 172.21.15.78:0/1598620695 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c44108a60 con 0x7f5c441075c0 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.272+0000 7f5c2b7fe700 1 -- 172.21.15.78:0/1598620695 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c340146a0 con 0x7f5c441075c0 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.272+0000 7f5c49332700 1 -- 172.21.15.78:0/1598620695 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c44108f60 con 0x7f5c441075c0 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.273+0000 7f5c49332700 1 -- 172.21.15.78:0/1598620695 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c44045ad0 con 0x7f5c441075c0 2023-12-21T03:56:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.273+0000 7f5c2b7fe700 1 -- 172.21.15.78:0/1598620695 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5c34013a80 con 0x7f5c441075c0 2023-12-21T03:56:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.273+0000 7f5c2b7fe700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5c2c05c2b0 0x7f5c2c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.273+0000 7f5c2b7fe700 1 -- 172.21.15.78:0/1598620695 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5c340770e0 con 0x7f5c441075c0 2023-12-21T03:56:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.273+0000 7f5c42ffd700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5c2c05c2b0 0x7f5c2c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.274+0000 7f5c42ffd700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5c2c05c2b0 0x7f5c2c05e770 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5c440f6650 tx=0x7f5c38008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.277+0000 7f5c2b7fe700 1 -- 172.21.15.78:0/1598620695 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c3404a0e0 con 0x7f5c441075c0 2023-12-21T03:56:42.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.484+0000 7f5c49332700 1 -- 172.21.15.78:0/1598620695 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5c440fde90 con 0x7f5c441075c0 2023-12-21T03:56:42.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.484+0000 7f5c2b7fe700 1 -- 172.21.15.78:0/1598620695 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f5c3404d700 con 0x7f5c441075c0 2023-12-21T03:56:42.485 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:42.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.486+0000 7f5c49332700 1 -- 172.21.15.78:0/1598620695 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5c2c05c2b0 msgr2=0x7f5c2c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:42.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.486+0000 7f5c49332700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5c2c05c2b0 0x7f5c2c05e770 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5c440f6650 tx=0x7f5c38008040).stop 2023-12-21T03:56:42.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.486+0000 7f5c49332700 1 -- 172.21.15.78:0/1598620695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c441075c0 msgr2=0x7f5c441084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:42.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.486+0000 7f5c49332700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c441075c0 0x7f5c441084b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f5c34009f50 tx=0x7f5c34009b30).stop 2023-12-21T03:56:42.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.486+0000 7f5c49332700 1 -- 172.21.15.78:0/1598620695 shutdown_connections 2023-12-21T03:56:42.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.486+0000 7f5c49332700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f5c2c05c2b0 0x7f5c2c05e770 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:42.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.486+0000 7f5c49332700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c440f5550 0x7f5c441070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:42.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.486+0000 7f5c49332700 1 --2- 172.21.15.78:0/1598620695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c441075c0 0x7f5c441084b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:42.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.486+0000 7f5c49332700 1 -- 172.21.15.78:0/1598620695 >> 172.21.15.78:0/1598620695 conn(0x7f5c440f0cb0 msgr2=0x7f5c440603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:42.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.486+0000 7f5c49332700 1 -- 172.21.15.78:0/1598620695 shutdown_connections 2023-12-21T03:56:42.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:42.486+0000 7f5c49332700 1 -- 172.21.15.78:0/1598620695 wait complete. 2023-12-21T03:56:42.872 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:41.691049+0000 mgr.smithi078.zidbgq (mgr.14162) 580 : cluster [DBG] pgmap v541: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:43.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:42.485409+0000 mon.smithi078 (mon.0) 582 : audit [DBG] from='client.? 172.21.15.78:0/1598620695' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:41.691049+0000 mgr.smithi078.zidbgq (mgr.14162) 580 : cluster [DBG] pgmap v541: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:43.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:42.485409+0000 mon.smithi078 (mon.0) 582 : audit [DBG] from='client.? 172.21.15.78:0/1598620695' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:45.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:43.691471+0000 mgr.smithi078.zidbgq (mgr.14162) 581 : cluster [DBG] pgmap v542: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:45.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:43.691471+0000 mgr.smithi078.zidbgq (mgr.14162) 581 : cluster [DBG] pgmap v542: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:45.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:46.116 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:47.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.136+0000 7f16e2dfd700 1 -- 172.21.15.78:0/3310150347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc0f7a60 msgr2=0x7f16dc0f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:47.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.136+0000 7f16e2dfd700 1 --2- 172.21.15.78:0/3310150347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc0f7a60 0x7f16dc0f7ec0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f16c8004660 tx=0x7f16c8009b30).stop 2023-12-21T03:56:47.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.136+0000 7f16e2dfd700 1 -- 172.21.15.78:0/3310150347 shutdown_connections 2023-12-21T03:56:47.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.136+0000 7f16e2dfd700 1 --2- 172.21.15.78:0/3310150347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc0f7a60 0x7f16dc0f7ec0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:47.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.136+0000 7f16e2dfd700 1 --2- 172.21.15.78:0/3310150347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16dc0f67c0 0x7f16dc0f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:47.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.136+0000 7f16e2dfd700 1 -- 172.21.15.78:0/3310150347 >> 172.21.15.78:0/3310150347 conn(0x7f16dc0f1f20 msgr2=0x7f16dc0f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:47.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.136+0000 7f16e2dfd700 1 -- 172.21.15.78:0/3310150347 shutdown_connections 2023-12-21T03:56:47.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.136+0000 7f16e2dfd700 1 -- 172.21.15.78:0/3310150347 wait complete. 2023-12-21T03:56:47.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.137+0000 7f16e2dfd700 1 Processor -- start 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.137+0000 7f16e2dfd700 1 -- start start 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.137+0000 7f16e2dfd700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16dc0f67c0 0x7f16dc1070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.137+0000 7f16e2dfd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc0f7a60 0x7f16dc1075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.137+0000 7f16e2dfd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc107b50 con 0x7f16dc0f7a60 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.137+0000 7f16e2dfd700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc107c90 con 0x7f16dc0f67c0 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.137+0000 7f16dbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc0f7a60 0x7f16dc1075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.137+0000 7f16dbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc0f7a60 0x7f16dc1075a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54966/0 (socket says 172.21.15.78:54966) 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.137+0000 7f16dbfff700 1 -- 172.21.15.78:0/2814290526 learned_addr learned my addr 172.21.15.78:0/2814290526 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.137+0000 7f16e0b99700 1 --2- 172.21.15.78:0/2814290526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16dc0f67c0 0x7f16dc1070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.138+0000 7f16dbfff700 1 -- 172.21.15.78:0/2814290526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16dc0f67c0 msgr2=0x7f16dc1070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.138+0000 7f16dbfff700 1 --2- 172.21.15.78:0/2814290526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16dc0f67c0 0x7f16dc1070b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:47.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.138+0000 7f16dbfff700 1 -- 172.21.15.78:0/2814290526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c8005040 con 0x7f16dc0f7a60 2023-12-21T03:56:47.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.138+0000 7f16dbfff700 1 --2- 172.21.15.78:0/2814290526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc0f7a60 0x7f16dc1075a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f16c8009830 tx=0x7f16c8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:47.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.138+0000 7f16d97fa700 1 -- 172.21.15.78:0/2814290526 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16c8018070 con 0x7f16dc0f7a60 2023-12-21T03:56:47.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.138+0000 7f16d97fa700 1 -- 172.21.15.78:0/2814290526 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16c8013410 con 0x7f16dc0f7a60 2023-12-21T03:56:47.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.138+0000 7f16e2dfd700 1 -- 172.21.15.78:0/2814290526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16dc1085d0 con 0x7f16dc0f7a60 2023-12-21T03:56:47.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.138+0000 7f16d97fa700 1 -- 172.21.15.78:0/2814290526 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16c801c400 con 0x7f16dc0f7a60 2023-12-21T03:56:47.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.138+0000 7f16e2dfd700 1 -- 172.21.15.78:0/2814290526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16dc108b30 con 0x7f16dc0f7a60 2023-12-21T03:56:47.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.139+0000 7f16e2dfd700 1 -- 172.21.15.78:0/2814290526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16dc045ad0 con 0x7f16dc0f7a60 2023-12-21T03:56:47.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.139+0000 7f16d97fa700 1 -- 172.21.15.78:0/2814290526 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f16c801d420 con 0x7f16dc0f7a60 2023-12-21T03:56:47.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.139+0000 7f16d97fa700 1 --2- 172.21.15.78:0/2814290526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f16cc05c2a0 0x7f16cc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:47.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.139+0000 7f16d97fa700 1 -- 172.21.15.78:0/2814290526 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f16c8077930 con 0x7f16dc0f7a60 2023-12-21T03:56:47.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.139+0000 7f16e0b99700 1 --2- 172.21.15.78:0/2814290526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f16cc05c2a0 0x7f16cc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:47.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.140+0000 7f16e0b99700 1 --2- 172.21.15.78:0/2814290526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f16cc05c2a0 0x7f16cc05e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f16dc0f78c0 tx=0x7f16d0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:47.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.143+0000 7f16d97fa700 1 -- 172.21.15.78:0/2814290526 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16c804a950 con 0x7f16dc0f7a60 2023-12-21T03:56:47.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.350+0000 7f16e2dfd700 1 -- 172.21.15.78:0/2814290526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f16dc027a00 con 0x7f16dc0f7a60 2023-12-21T03:56:47.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.351+0000 7f16d97fa700 1 -- 172.21.15.78:0/2814290526 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f16c804df70 con 0x7f16dc0f7a60 2023-12-21T03:56:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:47.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.352+0000 7f16e2dfd700 1 -- 172.21.15.78:0/2814290526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f16cc05c2a0 msgr2=0x7f16cc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:47.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.352+0000 7f16e2dfd700 1 --2- 172.21.15.78:0/2814290526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f16cc05c2a0 0x7f16cc05e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f16dc0f78c0 tx=0x7f16d0009040).stop 2023-12-21T03:56:47.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.352+0000 7f16e2dfd700 1 -- 172.21.15.78:0/2814290526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc0f7a60 msgr2=0x7f16dc1075a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:47.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.352+0000 7f16e2dfd700 1 --2- 172.21.15.78:0/2814290526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc0f7a60 0x7f16dc1075a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f16c8009830 tx=0x7f16c8009b30).stop 2023-12-21T03:56:47.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.352+0000 7f16e2dfd700 1 -- 172.21.15.78:0/2814290526 shutdown_connections 2023-12-21T03:56:47.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.352+0000 7f16e2dfd700 1 --2- 172.21.15.78:0/2814290526 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f16cc05c2a0 0x7f16cc05e760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:47.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.352+0000 7f16e2dfd700 1 --2- 172.21.15.78:0/2814290526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16dc0f67c0 0x7f16dc1070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:47.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.352+0000 7f16e2dfd700 1 --2- 172.21.15.78:0/2814290526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc0f7a60 0x7f16dc1075a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:47.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.352+0000 7f16e2dfd700 1 -- 172.21.15.78:0/2814290526 >> 172.21.15.78:0/2814290526 conn(0x7f16dc0f1f20 msgr2=0x7f16dc0fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:47.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.353+0000 7f16e2dfd700 1 -- 172.21.15.78:0/2814290526 shutdown_connections 2023-12-21T03:56:47.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:47.353+0000 7f16e2dfd700 1 -- 172.21.15.78:0/2814290526 wait complete. 2023-12-21T03:56:47.679 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:45.691890+0000 mgr.smithi078.zidbgq (mgr.14162) 582 : cluster [DBG] pgmap v543: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:47.679 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:47.352218+0000 mon.smithi078 (mon.0) 583 : audit [DBG] from='client.? 172.21.15.78:0/2814290526' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:47.698 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:45.691890+0000 mgr.smithi078.zidbgq (mgr.14162) 582 : cluster [DBG] pgmap v543: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:47.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:47.352218+0000 mon.smithi078 (mon.0) 583 : audit [DBG] from='client.? 172.21.15.78:0/2814290526' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:49.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:47.692244+0000 mgr.smithi078.zidbgq (mgr.14162) 583 : cluster [DBG] pgmap v544: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:49.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:47.692244+0000 mgr.smithi078.zidbgq (mgr.14162) 583 : cluster [DBG] pgmap v544: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:50.699 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:51.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:49.692598+0000 mgr.smithi078.zidbgq (mgr.14162) 584 : cluster [DBG] pgmap v545: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:51.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:49.692598+0000 mgr.smithi078.zidbgq (mgr.14162) 584 : cluster [DBG] pgmap v545: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:51.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.926+0000 7fad408e3700 1 -- 172.21.15.78:0/1992144291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0f6960 msgr2=0x7fad3c0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:51.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.926+0000 7fad408e3700 1 --2- 172.21.15.78:0/1992144291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0f6960 0x7fad3c0f6dc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fad2c004660 tx=0x7fad2c009b30).stop 2023-12-21T03:56:51.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.926+0000 7fad408e3700 1 -- 172.21.15.78:0/1992144291 shutdown_connections 2023-12-21T03:56:51.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.926+0000 7fad408e3700 1 --2- 172.21.15.78:0/1992144291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0f6960 0x7fad3c0f6dc0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:51.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.926+0000 7fad408e3700 1 --2- 172.21.15.78:0/1992144291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad3c0f56c0 0x7fad3c0f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:51.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.926+0000 7fad408e3700 1 -- 172.21.15.78:0/1992144291 >> 172.21.15.78:0/1992144291 conn(0x7fad3c0f0e20 msgr2=0x7fad3c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:51.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.926+0000 7fad408e3700 1 -- 172.21.15.78:0/1992144291 shutdown_connections 2023-12-21T03:56:51.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.926+0000 7fad408e3700 1 -- 172.21.15.78:0/1992144291 wait complete. 2023-12-21T03:56:51.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.927+0000 7fad408e3700 1 Processor -- start 2023-12-21T03:56:51.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.927+0000 7fad408e3700 1 -- start start 2023-12-21T03:56:51.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.927+0000 7fad408e3700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad3c0f56c0 0x7fad3c109de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:51.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.927+0000 7fad408e3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0f6960 0x7fad3c10c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:51.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.927+0000 7fad408e3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c060940 con 0x7fad3c0f6960 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.927+0000 7fad408e3700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c060a80 con 0x7fad3c0f56c0 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad39d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0f6960 0x7fad3c10c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad39d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0f6960 0x7fad3c10c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36142/0 (socket says 172.21.15.78:36142) 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad39d9b700 1 -- 172.21.15.78:0/700979683 learned_addr learned my addr 172.21.15.78:0/700979683 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad3a59c700 1 --2- 172.21.15.78:0/700979683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad3c0f56c0 0x7fad3c109de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad39d9b700 1 -- 172.21.15.78:0/700979683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad3c0f56c0 msgr2=0x7fad3c109de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad39d9b700 1 --2- 172.21.15.78:0/700979683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad3c0f56c0 0x7fad3c109de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad39d9b700 1 -- 172.21.15.78:0/700979683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c005040 con 0x7fad3c0f6960 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad39d9b700 1 --2- 172.21.15.78:0/700979683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0f6960 0x7fad3c10c2e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7fad2c009f50 tx=0x7fad2c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad32ffd700 1 -- 172.21.15.78:0/700979683 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad2c018070 con 0x7fad3c0f6960 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad32ffd700 1 -- 172.21.15.78:0/700979683 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad2c013410 con 0x7fad3c0f6960 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad408e3700 1 -- 172.21.15.78:0/700979683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c10c830 con 0x7fad3c0f6960 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad32ffd700 1 -- 172.21.15.78:0/700979683 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad2c01c3c0 con 0x7fad3c0f6960 2023-12-21T03:56:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.928+0000 7fad408e3700 1 -- 172.21.15.78:0/700979683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c10cd90 con 0x7fad3c0f6960 2023-12-21T03:56:51.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.929+0000 7fad408e3700 1 -- 172.21.15.78:0/700979683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c045ad0 con 0x7fad3c0f6960 2023-12-21T03:56:51.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.930+0000 7fad32ffd700 1 -- 172.21.15.78:0/700979683 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fad2c01d420 con 0x7fad3c0f6960 2023-12-21T03:56:51.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.930+0000 7fad32ffd700 1 --2- 172.21.15.78:0/700979683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad2805c2a0 0x7fad2805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:51.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.930+0000 7fad32ffd700 1 -- 172.21.15.78:0/700979683 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fad2c077ea0 con 0x7fad3c0f6960 2023-12-21T03:56:51.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.930+0000 7fad3a59c700 1 --2- 172.21.15.78:0/700979683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad2805c2a0 0x7fad2805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:51.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.930+0000 7fad3a59c700 1 --2- 172.21.15.78:0/700979683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad2805c2a0 0x7fad2805e760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fad3c0605c0 tx=0x7fad2400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:51.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:51.933+0000 7fad32ffd700 1 -- 172.21.15.78:0/700979683 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad2c04af90 con 0x7fad3c0f6960 2023-12-21T03:56:52.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.142+0000 7fad408e3700 1 -- 172.21.15.78:0/700979683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fad3c027a00 con 0x7fad3c0f6960 2023-12-21T03:56:52.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.142+0000 7fad32ffd700 1 -- 172.21.15.78:0/700979683 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fad2c04e5b0 con 0x7fad3c0f6960 2023-12-21T03:56:52.144 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:52.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.144+0000 7fad408e3700 1 -- 172.21.15.78:0/700979683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad2805c2a0 msgr2=0x7fad2805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:52.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.144+0000 7fad408e3700 1 --2- 172.21.15.78:0/700979683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad2805c2a0 0x7fad2805e760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fad3c0605c0 tx=0x7fad2400b040).stop 2023-12-21T03:56:52.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.144+0000 7fad408e3700 1 -- 172.21.15.78:0/700979683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0f6960 msgr2=0x7fad3c10c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:52.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.144+0000 7fad408e3700 1 --2- 172.21.15.78:0/700979683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0f6960 0x7fad3c10c2e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7fad2c009f50 tx=0x7fad2c009b30).stop 2023-12-21T03:56:52.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.144+0000 7fad408e3700 1 -- 172.21.15.78:0/700979683 shutdown_connections 2023-12-21T03:56:52.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.144+0000 7fad408e3700 1 --2- 172.21.15.78:0/700979683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fad2805c2a0 0x7fad2805e760 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:52.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.144+0000 7fad408e3700 1 --2- 172.21.15.78:0/700979683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad3c0f56c0 0x7fad3c109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:52.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.144+0000 7fad408e3700 1 --2- 172.21.15.78:0/700979683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0f6960 0x7fad3c10c2e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:52.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.144+0000 7fad408e3700 1 -- 172.21.15.78:0/700979683 >> 172.21.15.78:0/700979683 conn(0x7fad3c0f0e20 msgr2=0x7fad3c0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:52.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.144+0000 7fad408e3700 1 -- 172.21.15.78:0/700979683 shutdown_connections 2023-12-21T03:56:52.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:52.144+0000 7fad408e3700 1 -- 172.21.15.78:0/700979683 wait complete. 2023-12-21T03:56:52.521 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:52.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:52 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:52.143742+0000 mon.smithi078 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.78:0/700979683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:52.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:52 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:52.143742+0000 mon.smithi078 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.78:0/700979683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:53.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:51.692933+0000 mgr.smithi078.zidbgq (mgr.14162) 585 : cluster [DBG] pgmap v546: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:53.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:51.692933+0000 mgr.smithi078.zidbgq (mgr.14162) 585 : cluster [DBG] pgmap v546: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:55.521 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:56:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:56:55.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:53.693335+0000 mgr.smithi078.zidbgq (mgr.14162) 586 : cluster [DBG] pgmap v547: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:55.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:53.693335+0000 mgr.smithi078.zidbgq (mgr.14162) 586 : cluster [DBG] pgmap v547: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:56.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.701+0000 7f943ca6f700 1 -- 172.21.15.78:0/3511760187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94380f67b0 msgr2=0x7f94380f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:56.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.701+0000 7f943ca6f700 1 --2- 172.21.15.78:0/3511760187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94380f67b0 0x7f94380f6c10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f9428004660 tx=0x7f9428009b30).stop 2023-12-21T03:56:56.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.701+0000 7f943ca6f700 1 -- 172.21.15.78:0/3511760187 shutdown_connections 2023-12-21T03:56:56.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.701+0000 7f943ca6f700 1 --2- 172.21.15.78:0/3511760187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94380f67b0 0x7f94380f6c10 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:56.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.701+0000 7f943ca6f700 1 --2- 172.21.15.78:0/3511760187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94380f5510 0x7f94380f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:56.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.701+0000 7f943ca6f700 1 -- 172.21.15.78:0/3511760187 >> 172.21.15.78:0/3511760187 conn(0x7f94380f0c70 msgr2=0x7f94380f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:56.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.701+0000 7f943ca6f700 1 -- 172.21.15.78:0/3511760187 shutdown_connections 2023-12-21T03:56:56.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.701+0000 7f943ca6f700 1 -- 172.21.15.78:0/3511760187 wait complete. 2023-12-21T03:56:56.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.702+0000 7f943ca6f700 1 Processor -- start 2023-12-21T03:56:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.702+0000 7f943ca6f700 1 -- start start 2023-12-21T03:56:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.702+0000 7f943ca6f700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94380f5510 0x7f94381090a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.702+0000 7f943ca6f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94380f67b0 0x7f9438109590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.702+0000 7f943ca6f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9438109b40 con 0x7f94380f67b0 2023-12-21T03:56:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.702+0000 7f943ca6f700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9438109c80 con 0x7f94380f5510 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.702+0000 7f9436ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94380f67b0 0x7f9438109590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.702+0000 7f9436ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94380f67b0 0x7f9438109590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36164/0 (socket says 172.21.15.78:36164) 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.702+0000 7f9436ffd700 1 -- 172.21.15.78:0/2798190891 learned_addr learned my addr 172.21.15.78:0/2798190891 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.703+0000 7f94377fe700 1 --2- 172.21.15.78:0/2798190891 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94380f5510 0x7f94381090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.703+0000 7f9436ffd700 1 -- 172.21.15.78:0/2798190891 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94380f5510 msgr2=0x7f94381090a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.703+0000 7f9436ffd700 1 --2- 172.21.15.78:0/2798190891 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94380f5510 0x7f94381090a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.703+0000 7f9436ffd700 1 -- 172.21.15.78:0/2798190891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9428005040 con 0x7f94380f67b0 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.703+0000 7f9436ffd700 1 --2- 172.21.15.78:0/2798190891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94380f67b0 0x7f9438109590 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f9428009f50 tx=0x7f9428009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.703+0000 7f942ffff700 1 -- 172.21.15.78:0/2798190891 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9428018070 con 0x7f94380f67b0 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.703+0000 7f942ffff700 1 -- 172.21.15.78:0/2798190891 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9428013410 con 0x7f94380f67b0 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.703+0000 7f942ffff700 1 -- 172.21.15.78:0/2798190891 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f942801c3c0 con 0x7f94380f67b0 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.703+0000 7f943ca6f700 1 -- 172.21.15.78:0/2798190891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943810a5c0 con 0x7f94380f67b0 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.703+0000 7f943ca6f700 1 -- 172.21.15.78:0/2798190891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943810ab20 con 0x7f94380f67b0 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.704+0000 7f943ca6f700 1 -- 172.21.15.78:0/2798190891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9438045ad0 con 0x7f94380f67b0 2023-12-21T03:56:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.704+0000 7f942ffff700 1 -- 172.21.15.78:0/2798190891 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f942801d420 con 0x7f94380f67b0 2023-12-21T03:56:56.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.705+0000 7f942ffff700 1 --2- 172.21.15.78:0/2798190891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f942405c2b0 0x7f942405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:56:56.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.705+0000 7f942ffff700 1 -- 172.21.15.78:0/2798190891 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f94280777a0 con 0x7f94380f67b0 2023-12-21T03:56:56.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.705+0000 7f94377fe700 1 --2- 172.21.15.78:0/2798190891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f942405c2b0 0x7f942405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:56:56.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.705+0000 7f94377fe700 1 --2- 172.21.15.78:0/2798190891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f942405c2b0 0x7f942405e770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f94380f6610 tx=0x7f9420009c00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:56:56.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.709+0000 7f942ffff700 1 -- 172.21.15.78:0/2798190891 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f942804a820 con 0x7f94380f67b0 2023-12-21T03:56:56.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.916+0000 7f943ca6f700 1 -- 172.21.15.78:0/2798190891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f94380fdd50 con 0x7f94380f67b0 2023-12-21T03:56:56.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.917+0000 7f942ffff700 1 -- 172.21.15.78:0/2798190891 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f942804de40 con 0x7f94380f67b0 2023-12-21T03:56:56.919 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:56:56.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.918+0000 7f943ca6f700 1 -- 172.21.15.78:0/2798190891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f942405c2b0 msgr2=0x7f942405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:56.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.918+0000 7f943ca6f700 1 --2- 172.21.15.78:0/2798190891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f942405c2b0 0x7f942405e770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f94380f6610 tx=0x7f9420009c00).stop 2023-12-21T03:56:56.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.918+0000 7f943ca6f700 1 -- 172.21.15.78:0/2798190891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94380f67b0 msgr2=0x7f9438109590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:56:56.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.918+0000 7f943ca6f700 1 --2- 172.21.15.78:0/2798190891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94380f67b0 0x7f9438109590 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f9428009f50 tx=0x7f9428009b30).stop 2023-12-21T03:56:56.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.918+0000 7f943ca6f700 1 -- 172.21.15.78:0/2798190891 shutdown_connections 2023-12-21T03:56:56.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.918+0000 7f943ca6f700 1 --2- 172.21.15.78:0/2798190891 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f942405c2b0 0x7f942405e770 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:56.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.918+0000 7f943ca6f700 1 --2- 172.21.15.78:0/2798190891 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94380f5510 0x7f94381090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:56.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.918+0000 7f943ca6f700 1 --2- 172.21.15.78:0/2798190891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94380f67b0 0x7f9438109590 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:56:56.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.918+0000 7f943ca6f700 1 -- 172.21.15.78:0/2798190891 >> 172.21.15.78:0/2798190891 conn(0x7f94380f0c70 msgr2=0x7f9438060380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:56:56.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.918+0000 7f943ca6f700 1 -- 172.21.15.78:0/2798190891 shutdown_connections 2023-12-21T03:56:56.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:56:56.918+0000 7f943ca6f700 1 -- 172.21.15.78:0/2798190891 wait complete. 2023-12-21T03:56:57.296 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:56:57.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:55.693651+0000 mgr.smithi078.zidbgq (mgr.14162) 587 : cluster [DBG] pgmap v548: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:57.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:56:56.918118+0000 mon.smithi078 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.78:0/2798190891' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:55.693651+0000 mgr.smithi078.zidbgq (mgr.14162) 587 : cluster [DBG] pgmap v548: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:57.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:56:56.918118+0000 mon.smithi078 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.78:0/2798190891' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:56:59.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:56:59 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:57.694025+0000 mgr.smithi078.zidbgq (mgr.14162) 588 : cluster [DBG] pgmap v549: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:56:59.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:56:59 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:57.694025+0000 mgr.smithi078.zidbgq (mgr.14162) 588 : cluster [DBG] pgmap v549: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:00.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:00.533 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:01.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.500+0000 7faa52a2d700 1 -- 172.21.15.78:0/57622815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa4c0f6990 msgr2=0x7faa4c0f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:01.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.500+0000 7faa52a2d700 1 --2- 172.21.15.78:0/57622815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa4c0f6990 0x7faa4c0f6df0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7faa40004660 tx=0x7faa40009b30).stop 2023-12-21T03:57:01.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.500+0000 7faa52a2d700 1 -- 172.21.15.78:0/57622815 shutdown_connections 2023-12-21T03:57:01.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.500+0000 7faa52a2d700 1 --2- 172.21.15.78:0/57622815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa4c0f6990 0x7faa4c0f6df0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:01.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.500+0000 7faa52a2d700 1 --2- 172.21.15.78:0/57622815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa4c0f56f0 0x7faa4c0f5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:57:01.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.500+0000 7faa52a2d700 1 -- 172.21.15.78:0/57622815 >> 172.21.15.78:0/57622815 conn(0x7faa4c0f0e50 msgr2=0x7faa4c0f3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:01.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.500+0000 7faa52a2d700 1 -- 172.21.15.78:0/57622815 shutdown_connections 2023-12-21T03:57:01.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.500+0000 7faa52a2d700 1 -- 172.21.15.78:0/57622815 wait complete. 2023-12-21T03:57:01.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.501+0000 7faa52a2d700 1 Processor -- start 2023-12-21T03:57:01.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.501+0000 7faa52a2d700 1 -- start start 2023-12-21T03:57:01.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa52a2d700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa4c0f56f0 0x7faa4c109db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:01.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa52a2d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa4c0f6990 0x7faa4c10c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:01.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa52a2d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c0609d0 con 0x7faa4c0f6990 2023-12-21T03:57:01.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa52a2d700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c060b10 con 0x7faa4c0f56f0 2023-12-21T03:57:01.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa4b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa4c0f6990 0x7faa4c10c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:01.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa4b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa4c0f6990 0x7faa4c10c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55700/0 (socket says 172.21.15.78:55700) 2023-12-21T03:57:01.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa4b7fe700 1 -- 172.21.15.78:0/2395271666 learned_addr learned my addr 172.21.15.78:0/2395271666 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:01.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa4bfff700 1 --2- 172.21.15.78:0/2395271666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa4c0f56f0 0x7faa4c109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:01.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa4b7fe700 1 -- 172.21.15.78:0/2395271666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa4c0f56f0 msgr2=0x7faa4c109db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:01.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa4b7fe700 1 --2- 172.21.15.78:0/2395271666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa4c0f56f0 0x7faa4c109db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:01.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa4b7fe700 1 -- 172.21.15.78:0/2395271666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40005040 con 0x7faa4c0f6990 2023-12-21T03:57:01.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.502+0000 7faa4b7fe700 1 --2- 172.21.15.78:0/2395271666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa4c0f6990 0x7faa4c10c2b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7faa40009830 tx=0x7faa40009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:01.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.503+0000 7faa48ff9700 1 -- 172.21.15.78:0/2395271666 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa4001c070 con 0x7faa4c0f6990 2023-12-21T03:57:01.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.503+0000 7faa48ff9700 1 -- 172.21.15.78:0/2395271666 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa40012a80 con 0x7faa4c0f6990 2023-12-21T03:57:01.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.503+0000 7faa48ff9700 1 -- 172.21.15.78:0/2395271666 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa40021910 con 0x7faa4c0f6990 2023-12-21T03:57:01.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.503+0000 7faa52a2d700 1 -- 172.21.15.78:0/2395271666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4c105b90 con 0x7faa4c0f6990 2023-12-21T03:57:01.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.503+0000 7faa52a2d700 1 -- 172.21.15.78:0/2395271666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4c1060f0 con 0x7faa4c0f6990 2023-12-21T03:57:01.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.504+0000 7faa52a2d700 1 -- 172.21.15.78:0/2395271666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa4c045ad0 con 0x7faa4c0f6990 2023-12-21T03:57:01.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.504+0000 7faa48ff9700 1 -- 172.21.15.78:0/2395271666 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7faa40012bf0 con 0x7faa4c0f6990 2023-12-21T03:57:01.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.504+0000 7faa48ff9700 1 --2- 172.21.15.78:0/2395271666 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faa3405c2a0 0x7faa3405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:01.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.505+0000 7faa48ff9700 1 -- 172.21.15.78:0/2395271666 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7faa4007be00 con 0x7faa4c0f6990 2023-12-21T03:57:01.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.505+0000 7faa4bfff700 1 --2- 172.21.15.78:0/2395271666 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faa3405c2a0 0x7faa3405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:01.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.505+0000 7faa4bfff700 1 --2- 172.21.15.78:0/2395271666 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faa3405c2a0 0x7faa3405e760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7faa4c0605c0 tx=0x7faa3c008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:01.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.508+0000 7faa48ff9700 1 -- 172.21.15.78:0/2395271666 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa4004ee20 con 0x7faa4c0f6990 2023-12-21T03:57:01.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.723+0000 7faa52a2d700 1 -- 172.21.15.78:0/2395271666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7faa4c027a00 con 0x7faa4c0f6990 2023-12-21T03:57:01.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.723+0000 7faa48ff9700 1 -- 172.21.15.78:0/2395271666 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7faa40052440 con 0x7faa4c0f6990 2023-12-21T03:57:01.725 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.725+0000 7faa52a2d700 1 -- 172.21.15.78:0/2395271666 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faa3405c2a0 msgr2=0x7faa3405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.725+0000 7faa52a2d700 1 --2- 172.21.15.78:0/2395271666 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faa3405c2a0 0x7faa3405e760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7faa4c0605c0 tx=0x7faa3c008040).stop 2023-12-21T03:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.725+0000 7faa52a2d700 1 -- 172.21.15.78:0/2395271666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa4c0f6990 msgr2=0x7faa4c10c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.725+0000 7faa52a2d700 1 --2- 172.21.15.78:0/2395271666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa4c0f6990 0x7faa4c10c2b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7faa40009830 tx=0x7faa40009b30).stop 2023-12-21T03:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.725+0000 7faa52a2d700 1 -- 172.21.15.78:0/2395271666 shutdown_connections 2023-12-21T03:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.725+0000 7faa52a2d700 1 --2- 172.21.15.78:0/2395271666 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7faa3405c2a0 0x7faa3405e760 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.725+0000 7faa52a2d700 1 --2- 172.21.15.78:0/2395271666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa4c0f56f0 0x7faa4c109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.725+0000 7faa52a2d700 1 --2- 172.21.15.78:0/2395271666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa4c0f6990 0x7faa4c10c2b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.725+0000 7faa52a2d700 1 -- 172.21.15.78:0/2395271666 >> 172.21.15.78:0/2395271666 conn(0x7faa4c0f0e50 msgr2=0x7faa4c0fa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.725+0000 7faa52a2d700 1 -- 172.21.15.78:0/2395271666 shutdown_connections 2023-12-21T03:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:01.725+0000 7faa52a2d700 1 -- 172.21.15.78:0/2395271666 wait complete. 2023-12-21T03:57:01.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:56:59.694507+0000 mgr.smithi078.zidbgq (mgr.14162) 589 : cluster [DBG] pgmap v550: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:01.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:56:59.694507+0000 mgr.smithi078.zidbgq (mgr.14162) 589 : cluster [DBG] pgmap v550: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:02.151 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:02.766 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:02 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:01.724597+0000 mon.smithi078 (mon.0) 586 : audit [DBG] from='client.? 172.21.15.78:0/2395271666' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:02.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:02 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:01.724597+0000 mon.smithi078 (mon.0) 586 : audit [DBG] from='client.? 172.21.15.78:0/2395271666' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:03.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:03 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:01.694882+0000 mgr.smithi078.zidbgq (mgr.14162) 590 : cluster [DBG] pgmap v551: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:03.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:01.694882+0000 mgr.smithi078.zidbgq (mgr.14162) 590 : cluster [DBG] pgmap v551: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:05.152 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:05.393 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:05.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:05 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:03.695335+0000 mgr.smithi078.zidbgq (mgr.14162) 591 : cluster [DBG] pgmap v552: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:05.774 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:05 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:03.695335+0000 mgr.smithi078.zidbgq (mgr.14162) 591 : cluster [DBG] pgmap v552: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:06.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.453+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1743713728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9abc0f7a80 msgr2=0x7f9abc0f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:06.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.453+0000 7f9ac35b9700 1 --2- 172.21.15.78:0/1743713728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9abc0f7a80 0x7f9abc0f7ee0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f9ab8004660 tx=0x7f9ab8009b30).stop 2023-12-21T03:57:06.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.453+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1743713728 shutdown_connections 2023-12-21T03:57:06.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.453+0000 7f9ac35b9700 1 --2- 172.21.15.78:0/1743713728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9abc0f7a80 0x7f9abc0f7ee0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:06.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.453+0000 7f9ac35b9700 1 --2- 172.21.15.78:0/1743713728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9abc0f67e0 0x7f9abc0f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:06.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.453+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1743713728 >> 172.21.15.78:0/1743713728 conn(0x7f9abc0f1f80 msgr2=0x7f9abc0f43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.453+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1743713728 shutdown_connections 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.453+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1743713728 wait complete. 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.454+0000 7f9ac35b9700 1 Processor -- start 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.454+0000 7f9ac35b9700 1 -- start start 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.454+0000 7f9ac35b9700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9abc0f67e0 0x7f9abc107030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.454+0000 7f9ac35b9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9abc0f7a80 0x7f9abc107520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.454+0000 7f9ac35b9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc107ad0 con 0x7f9abc0f7a80 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.454+0000 7f9ac35b9700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc107c10 con 0x7f9abc0f67e0 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.454+0000 7f9ac0b54700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9abc0f7a80 0x7f9abc107520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.454+0000 7f9ac0b54700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9abc0f7a80 0x7f9abc107520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55722/0 (socket says 172.21.15.78:55722) 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.454+0000 7f9ac0b54700 1 -- 172.21.15.78:0/1610895976 learned_addr learned my addr 172.21.15.78:0/1610895976 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.454+0000 7f9ac1355700 1 --2- 172.21.15.78:0/1610895976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9abc0f67e0 0x7f9abc107030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.455+0000 7f9ac0b54700 1 -- 172.21.15.78:0/1610895976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9abc0f67e0 msgr2=0x7f9abc107030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.455+0000 7f9ac0b54700 1 --2- 172.21.15.78:0/1610895976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9abc0f67e0 0x7f9abc107030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.455+0000 7f9ac0b54700 1 -- 172.21.15.78:0/1610895976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8005040 con 0x7f9abc0f7a80 2023-12-21T03:57:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.455+0000 7f9ac0b54700 1 --2- 172.21.15.78:0/1610895976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9abc0f7a80 0x7f9abc107520 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f9ab8009f50 tx=0x7f9ab8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.455+0000 7f9ab1ffb700 1 -- 172.21.15.78:0/1610895976 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ab8018070 con 0x7f9abc0f7a80 2023-12-21T03:57:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.455+0000 7f9ab1ffb700 1 -- 172.21.15.78:0/1610895976 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ab8013410 con 0x7f9abc0f7a80 2023-12-21T03:57:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.455+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1610895976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc108550 con 0x7f9abc0f7a80 2023-12-21T03:57:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.455+0000 7f9ab1ffb700 1 -- 172.21.15.78:0/1610895976 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ab801c3c0 con 0x7f9abc0f7a80 2023-12-21T03:57:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.455+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1610895976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc108ab0 con 0x7f9abc0f7a80 2023-12-21T03:57:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.456+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1610895976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9abc045ad0 con 0x7f9abc0f7a80 2023-12-21T03:57:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.456+0000 7f9ab1ffb700 1 -- 172.21.15.78:0/1610895976 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9ab801d420 con 0x7f9abc0f7a80 2023-12-21T03:57:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.457+0000 7f9ab1ffb700 1 --2- 172.21.15.78:0/1610895976 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f9aa805c2a0 0x7f9aa805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.457+0000 7f9ab1ffb700 1 -- 172.21.15.78:0/1610895976 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f9ab8077800 con 0x7f9abc0f7a80 2023-12-21T03:57:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.457+0000 7f9ac1355700 1 --2- 172.21.15.78:0/1610895976 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f9aa805c2a0 0x7f9aa805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.457+0000 7f9ac1355700 1 --2- 172.21.15.78:0/1610895976 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f9aa805c2a0 0x7f9aa805e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f9abc0f78e0 tx=0x7f9aac00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:06.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.461+0000 7f9ab1ffb700 1 -- 172.21.15.78:0/1610895976 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ab804a800 con 0x7f9abc0f7a80 2023-12-21T03:57:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.668+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1610895976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9abc027a00 con 0x7f9abc0f7a80 2023-12-21T03:57:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.668+0000 7f9ab1ffb700 1 -- 172.21.15.78:0/1610895976 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f9ab804de20 con 0x7f9abc0f7a80 2023-12-21T03:57:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.670+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1610895976 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f9aa805c2a0 msgr2=0x7f9aa805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.670+0000 7f9ac35b9700 1 --2- 172.21.15.78:0/1610895976 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f9aa805c2a0 0x7f9aa805e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f9abc0f78e0 tx=0x7f9aac00b040).stop 2023-12-21T03:57:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.670+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1610895976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9abc0f7a80 msgr2=0x7f9abc107520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.670+0000 7f9ac35b9700 1 --2- 172.21.15.78:0/1610895976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9abc0f7a80 0x7f9abc107520 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f9ab8009f50 tx=0x7f9ab8009b30).stop 2023-12-21T03:57:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.670+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1610895976 shutdown_connections 2023-12-21T03:57:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.670+0000 7f9ac35b9700 1 --2- 172.21.15.78:0/1610895976 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f9aa805c2a0 0x7f9aa805e760 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.670+0000 7f9ac35b9700 1 --2- 172.21.15.78:0/1610895976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9abc0f67e0 0x7f9abc107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.670+0000 7f9ac35b9700 1 --2- 172.21.15.78:0/1610895976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9abc0f7a80 0x7f9abc107520 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.670+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1610895976 >> 172.21.15.78:0/1610895976 conn(0x7f9abc0f1f80 msgr2=0x7f9abc0fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.670+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1610895976 shutdown_connections 2023-12-21T03:57:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:06.670+0000 7f9ac35b9700 1 -- 172.21.15.78:0/1610895976 wait complete. 2023-12-21T03:57:07.062 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:05.695691+0000 mgr.smithi078.zidbgq (mgr.14162) 592 : cluster [DBG] pgmap v553: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:07.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:07 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:06.669559+0000 mon.smithi078 (mon.0) 587 : audit [DBG] from='client.? 172.21.15.78:0/1610895976' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:08.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:05.695691+0000 mgr.smithi078.zidbgq (mgr.14162) 592 : cluster [DBG] pgmap v553: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:08.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:06.669559+0000 mon.smithi078 (mon.0) 587 : audit [DBG] from='client.? 172.21.15.78:0/1610895976' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:09.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:09 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:07.696089+0000 mgr.smithi078.zidbgq (mgr.14162) 593 : cluster [DBG] pgmap v554: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:10.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:09 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:07.696089+0000 mgr.smithi078.zidbgq (mgr.14162) 593 : cluster [DBG] pgmap v554: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:10.064 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:10.306 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:11.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.324+0000 7fefaa9f4700 1 -- 172.21.15.78:0/2657341909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefa40f56c0 msgr2=0x7fefa40f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:11.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.324+0000 7fefaa9f4700 1 --2- 172.21.15.78:0/2657341909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefa40f56c0 0x7fefa40f5ae0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7fef94004660 tx=0x7fef94009b30).stop 2023-12-21T03:57:11.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.324+0000 7fefaa9f4700 1 -- 172.21.15.78:0/2657341909 shutdown_connections 2023-12-21T03:57:11.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.324+0000 7fefaa9f4700 1 --2- 172.21.15.78:0/2657341909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefa40f6960 0x7fefa40f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.324+0000 7fefaa9f4700 1 --2- 172.21.15.78:0/2657341909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefa40f56c0 0x7fefa40f5ae0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.324+0000 7fefaa9f4700 1 -- 172.21.15.78:0/2657341909 >> 172.21.15.78:0/2657341909 conn(0x7fefa40f0e20 msgr2=0x7fefa40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.324+0000 7fefaa9f4700 1 -- 172.21.15.78:0/2657341909 shutdown_connections 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.324+0000 7fefaa9f4700 1 -- 172.21.15.78:0/2657341909 wait complete. 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.324+0000 7fefaa9f4700 1 Processor -- start 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefaa9f4700 1 -- start start 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefaa9f4700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefa40f56c0 0x7fefa40f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefaa9f4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefa40f6960 0x7fefa40f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefaa9f4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40f8290 con 0x7fefa40f6960 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefaa9f4700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40f83d0 con 0x7fefa40f56c0 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefa37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefa40f6960 0x7fefa40f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefa37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefa40f6960 0x7fefa40f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57874/0 (socket says 172.21.15.78:57874) 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefa37fe700 1 -- 172.21.15.78:0/255997804 learned_addr learned my addr 172.21.15.78:0/255997804 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefa3fff700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefa40f56c0 0x7fefa40f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefa37fe700 1 -- 172.21.15.78:0/255997804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefa40f56c0 msgr2=0x7fefa40f9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:11.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefa37fe700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefa40f56c0 0x7fefa40f9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:11.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefa37fe700 1 -- 172.21.15.78:0/255997804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef94005040 con 0x7fefa40f6960 2023-12-21T03:57:11.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefa3fff700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefa40f56c0 0x7fefa40f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:57:11.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefa37fe700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefa40f6960 0x7fefa40f7ca0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fef9800b410 tx=0x7fef98006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:11.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.325+0000 7fefa0ff9700 1 -- 172.21.15.78:0/255997804 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef980125f0 con 0x7fefa40f6960 2023-12-21T03:57:11.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.326+0000 7fefaa9f4700 1 -- 172.21.15.78:0/255997804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa40f86b0 con 0x7fefa40f6960 2023-12-21T03:57:11.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.326+0000 7fefa0ff9700 1 -- 172.21.15.78:0/255997804 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef98012c00 con 0x7fefa40f6960 2023-12-21T03:57:11.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.326+0000 7fefa0ff9700 1 -- 172.21.15.78:0/255997804 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef98002aa0 con 0x7fefa40f6960 2023-12-21T03:57:11.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.326+0000 7fefaa9f4700 1 -- 172.21.15.78:0/255997804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa410f110 con 0x7fefa40f6960 2023-12-21T03:57:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.326+0000 7fefaa9f4700 1 -- 172.21.15.78:0/255997804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa4045ad0 con 0x7fefa40f6960 2023-12-21T03:57:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.327+0000 7fefa0ff9700 1 -- 172.21.15.78:0/255997804 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fef9800f030 con 0x7fefa40f6960 2023-12-21T03:57:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.327+0000 7fefa0ff9700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fef8c05c2a0 0x7fef8c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.327+0000 7fefa0ff9700 1 -- 172.21.15.78:0/255997804 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fef98076060 con 0x7fefa40f6960 2023-12-21T03:57:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.327+0000 7fefa3fff700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fef8c05c2a0 0x7fef8c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.328+0000 7fefa3fff700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fef8c05c2a0 0x7fef8c05e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fefa40f95b0 tx=0x7fef94009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.331+0000 7fefa0ff9700 1 -- 172.21.15.78:0/255997804 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef98049000 con 0x7fefa40f6960 2023-12-21T03:57:11.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.540+0000 7fefaa9f4700 1 -- 172.21.15.78:0/255997804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fefa40fded0 con 0x7fefa40f6960 2023-12-21T03:57:11.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.540+0000 7fefa0ff9700 1 -- 172.21.15.78:0/255997804 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fef9804c620 con 0x7fefa40f6960 2023-12-21T03:57:11.543 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.542+0000 7fefaa9f4700 1 -- 172.21.15.78:0/255997804 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fef8c05c2a0 msgr2=0x7fef8c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.542+0000 7fefaa9f4700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fef8c05c2a0 0x7fef8c05e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fefa40f95b0 tx=0x7fef94009b30).stop 2023-12-21T03:57:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.542+0000 7fefaa9f4700 1 -- 172.21.15.78:0/255997804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefa40f6960 msgr2=0x7fefa40f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.542+0000 7fefaa9f4700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefa40f6960 0x7fefa40f7ca0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fef9800b410 tx=0x7fef98006ef0).stop 2023-12-21T03:57:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.542+0000 7fefaa9f4700 1 -- 172.21.15.78:0/255997804 shutdown_connections 2023-12-21T03:57:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.542+0000 7fefaa9f4700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fef8c05c2a0 0x7fef8c05e760 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.542+0000 7fefaa9f4700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefa40f56c0 0x7fefa40f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.542+0000 7fefaa9f4700 1 --2- 172.21.15.78:0/255997804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefa40f6960 0x7fefa40f7ca0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.542+0000 7fefaa9f4700 1 -- 172.21.15.78:0/255997804 >> 172.21.15.78:0/255997804 conn(0x7fefa40f0e20 msgr2=0x7fefa40603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.542+0000 7fefaa9f4700 1 -- 172.21.15.78:0/255997804 shutdown_connections 2023-12-21T03:57:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:11.542+0000 7fefaa9f4700 1 -- 172.21.15.78:0/255997804 wait complete. 2023-12-21T03:57:11.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:11 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:09.696432+0000 mgr.smithi078.zidbgq (mgr.14162) 594 : cluster [DBG] pgmap v555: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:11.909 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:12.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:11 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:09.696432+0000 mgr.smithi078.zidbgq (mgr.14162) 594 : cluster [DBG] pgmap v555: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:12.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:12 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:11.541898+0000 mon.smithi078 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.78:0/255997804' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:13.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:12 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:11.541898+0000 mon.smithi078 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.78:0/255997804' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:13.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:13 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:11.696764+0000 mgr.smithi078.zidbgq (mgr.14162) 595 : cluster [DBG] pgmap v556: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:14.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:13 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:11.696764+0000 mgr.smithi078.zidbgq (mgr.14162) 595 : cluster [DBG] pgmap v556: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:14.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:15.154 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:15.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:15 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:13.697194+0000 mgr.smithi078.zidbgq (mgr.14162) 596 : cluster [DBG] pgmap v557: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:16.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:15 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:13.697194+0000 mgr.smithi078.zidbgq (mgr.14162) 596 : cluster [DBG] pgmap v557: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.127+0000 7fa4afc7f700 1 -- 172.21.15.78:0/3747470079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4a80f56c0 msgr2=0x7fa4a80f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.127+0000 7fa4afc7f700 1 --2- 172.21.15.78:0/3747470079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4a80f56c0 0x7fa4a80f5ae0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7fa498004660 tx=0x7fa498009b30).stop 2023-12-21T03:57:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.127+0000 7fa4afc7f700 1 -- 172.21.15.78:0/3747470079 shutdown_connections 2023-12-21T03:57:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.127+0000 7fa4afc7f700 1 --2- 172.21.15.78:0/3747470079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4a80f6960 0x7fa4a80f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.127+0000 7fa4afc7f700 1 --2- 172.21.15.78:0/3747470079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4a80f56c0 0x7fa4a80f5ae0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.127+0000 7fa4afc7f700 1 -- 172.21.15.78:0/3747470079 >> 172.21.15.78:0/3747470079 conn(0x7fa4a80f0e20 msgr2=0x7fa4a80f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.127+0000 7fa4afc7f700 1 -- 172.21.15.78:0/3747470079 shutdown_connections 2023-12-21T03:57:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.127+0000 7fa4afc7f700 1 -- 172.21.15.78:0/3747470079 wait complete. 2023-12-21T03:57:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.127+0000 7fa4afc7f700 1 Processor -- start 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.127+0000 7fa4afc7f700 1 -- start start 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4afc7f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4a80f56c0 0x7fa4a81070f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4afc7f700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4a80f6960 0x7fa4a81075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4afc7f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8107b90 con 0x7fa4a80f56c0 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4afc7f700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8107cd0 con 0x7fa4a80f6960 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4ada1b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4a80f56c0 0x7fa4a81070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4ada1b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4a80f56c0 0x7fa4a81070f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57902/0 (socket says 172.21.15.78:57902) 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4ada1b700 1 -- 172.21.15.78:0/2560285107 learned_addr learned my addr 172.21.15.78:0/2560285107 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4ada1b700 1 -- 172.21.15.78:0/2560285107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4a80f6960 msgr2=0x7fa4a81075e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4ada1b700 1 --2- 172.21.15.78:0/2560285107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4a80f6960 0x7fa4a81075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4ada1b700 1 -- 172.21.15.78:0/2560285107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498005040 con 0x7fa4a80f56c0 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4ada1b700 1 --2- 172.21.15.78:0/2560285107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4a80f56c0 0x7fa4a81070f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fa498004e30 tx=0x7fa498009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa49e7fc700 1 -- 172.21.15.78:0/2560285107 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa498018070 con 0x7fa4a80f56c0 2023-12-21T03:57:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa49e7fc700 1 -- 172.21.15.78:0/2560285107 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa498013410 con 0x7fa4a80f56c0 2023-12-21T03:57:16.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4afc7f700 1 -- 172.21.15.78:0/2560285107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a8108610 con 0x7fa4a80f56c0 2023-12-21T03:57:16.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa49e7fc700 1 -- 172.21.15.78:0/2560285107 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa49801c3c0 con 0x7fa4a80f56c0 2023-12-21T03:57:16.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.128+0000 7fa4afc7f700 1 -- 172.21.15.78:0/2560285107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a8108b70 con 0x7fa4a80f56c0 2023-12-21T03:57:16.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.129+0000 7fa4afc7f700 1 -- 172.21.15.78:0/2560285107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a8045ad0 con 0x7fa4a80f56c0 2023-12-21T03:57:16.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.129+0000 7fa49e7fc700 1 -- 172.21.15.78:0/2560285107 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa49801d420 con 0x7fa4a80f56c0 2023-12-21T03:57:16.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.130+0000 7fa49e7fc700 1 --2- 172.21.15.78:0/2560285107 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa49405c2a0 0x7fa49405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:16.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.130+0000 7fa49e7fc700 1 -- 172.21.15.78:0/2560285107 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa498077820 con 0x7fa4a80f56c0 2023-12-21T03:57:16.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.130+0000 7fa4ad21a700 1 --2- 172.21.15.78:0/2560285107 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa49405c2a0 0x7fa49405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:16.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.130+0000 7fa4ad21a700 1 --2- 172.21.15.78:0/2560285107 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa49405c2a0 0x7fa49405e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fa4a4004a10 tx=0x7fa4a40099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:16.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.134+0000 7fa49e7fc700 1 -- 172.21.15.78:0/2560285107 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa49804a820 con 0x7fa4a80f56c0 2023-12-21T03:57:16.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.341+0000 7fa4afc7f700 1 -- 172.21.15.78:0/2560285107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa4a8109270 con 0x7fa4a80f56c0 2023-12-21T03:57:16.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.342+0000 7fa49e7fc700 1 -- 172.21.15.78:0/2560285107 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fa49804de40 con 0x7fa4a80f56c0 2023-12-21T03:57:16.345 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:16.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.343+0000 7fa4afc7f700 1 -- 172.21.15.78:0/2560285107 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa49405c2a0 msgr2=0x7fa49405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:16.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.343+0000 7fa4afc7f700 1 --2- 172.21.15.78:0/2560285107 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa49405c2a0 0x7fa49405e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fa4a4004a10 tx=0x7fa4a40099e0).stop 2023-12-21T03:57:16.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.343+0000 7fa4afc7f700 1 -- 172.21.15.78:0/2560285107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4a80f56c0 msgr2=0x7fa4a81070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:16.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.343+0000 7fa4afc7f700 1 --2- 172.21.15.78:0/2560285107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4a80f56c0 0x7fa4a81070f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fa498004e30 tx=0x7fa498009b30).stop 2023-12-21T03:57:16.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.343+0000 7fa4afc7f700 1 -- 172.21.15.78:0/2560285107 shutdown_connections 2023-12-21T03:57:16.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.343+0000 7fa4afc7f700 1 --2- 172.21.15.78:0/2560285107 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa49405c2a0 0x7fa49405e760 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:16.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.343+0000 7fa4afc7f700 1 --2- 172.21.15.78:0/2560285107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4a80f56c0 0x7fa4a81070f0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:16.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.343+0000 7fa4afc7f700 1 --2- 172.21.15.78:0/2560285107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4a80f6960 0x7fa4a81075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-21T03:57:16.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.343+0000 7fa4afc7f700 1 -- 172.21.15.78:0/2560285107 >> 172.21.15.78:0/2560285107 conn(0x7fa4a80f0e20 msgr2=0x7fa4a80603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:16.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.343+0000 7fa4afc7f700 1 -- 172.21.15.78:0/2560285107 shutdown_connections 2023-12-21T03:57:16.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:16.343+0000 7fa4afc7f700 1 -- 172.21.15.78:0/2560285107 wait complete. 2023-12-21T03:57:16.700 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:16 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:16.343072+0000 mon.smithi078 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.78:0/2560285107' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:16.743 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:17.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:16 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:16.343072+0000 mon.smithi078 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.78:0/2560285107' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:17.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:17 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:15.697481+0000 mgr.smithi078.zidbgq (mgr.14162) 597 : cluster [DBG] pgmap v558: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:18.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:17 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:15.697481+0000 mgr.smithi078.zidbgq (mgr.14162) 597 : cluster [DBG] pgmap v558: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:19.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:19.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:19 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:17.697871+0000 mgr.smithi078.zidbgq (mgr.14162) 598 : cluster [DBG] pgmap v559: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:19.982 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:20.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:19 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:17.697871+0000 mgr.smithi078.zidbgq (mgr.14162) 598 : cluster [DBG] pgmap v559: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:21.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.076+0000 7f9635aa1700 1 -- 172.21.15.78:0/1961527011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96300f5510 msgr2=0x7f96300f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.076+0000 7f9635aa1700 1 --2- 172.21.15.78:0/1961527011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96300f5510 0x7f96300f5930 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f9624004660 tx=0x7f9624009b30).stop 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.076+0000 7f9635aa1700 1 -- 172.21.15.78:0/1961527011 shutdown_connections 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.076+0000 7f9635aa1700 1 --2- 172.21.15.78:0/1961527011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96300f67b0 0x7f96300f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.076+0000 7f9635aa1700 1 --2- 172.21.15.78:0/1961527011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96300f5510 0x7f96300f5930 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.076+0000 7f9635aa1700 1 -- 172.21.15.78:0/1961527011 >> 172.21.15.78:0/1961527011 conn(0x7f96300f0c70 msgr2=0x7f96300f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.076+0000 7f9635aa1700 1 -- 172.21.15.78:0/1961527011 shutdown_connections 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.077+0000 7f9635aa1700 1 -- 172.21.15.78:0/1961527011 wait complete. 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.077+0000 7f9635aa1700 1 Processor -- start 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.077+0000 7f9635aa1700 1 -- start start 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.077+0000 7f9635aa1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96300f5510 0x7f96300f9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.077+0000 7f9635aa1700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96300f67b0 0x7f96300f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.077+0000 7f9635aa1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300f8050 con 0x7f96300f5510 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.077+0000 7f9635aa1700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300f8190 con 0x7f96300f67b0 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.077+0000 7f9634a9f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96300f5510 0x7f96300f9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f9634a9f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96300f5510 0x7f96300f9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49952/0 (socket says 172.21.15.78:49952) 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f9634a9f700 1 -- 172.21.15.78:0/1255982155 learned_addr learned my addr 172.21.15.78:0/1255982155 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f962ffff700 1 --2- 172.21.15.78:0/1255982155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96300f67b0 0x7f96300f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f9634a9f700 1 -- 172.21.15.78:0/1255982155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96300f67b0 msgr2=0x7f96300f7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f9634a9f700 1 --2- 172.21.15.78:0/1255982155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96300f67b0 0x7f96300f7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f9634a9f700 1 -- 172.21.15.78:0/1255982155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9624005040 con 0x7f96300f5510 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f9634a9f700 1 --2- 172.21.15.78:0/1255982155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96300f5510 0x7f96300f9670 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f9624004e30 tx=0x7f9624009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f962d7fa700 1 -- 172.21.15.78:0/1255982155 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9624018070 con 0x7f96300f5510 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f9635aa1700 1 -- 172.21.15.78:0/1255982155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96300f8410 con 0x7f96300f5510 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f962d7fa700 1 -- 172.21.15.78:0/1255982155 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9624013410 con 0x7f96300f5510 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f962d7fa700 1 -- 172.21.15.78:0/1255982155 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f962401de20 con 0x7f96300f5510 2023-12-21T03:57:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.078+0000 7f9635aa1700 1 -- 172.21.15.78:0/1255982155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96300f8970 con 0x7f96300f5510 2023-12-21T03:57:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.079+0000 7f9635aa1700 1 -- 172.21.15.78:0/1255982155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9630045ad0 con 0x7f96300f5510 2023-12-21T03:57:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.079+0000 7f962d7fa700 1 -- 172.21.15.78:0/1255982155 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f962401c4a0 con 0x7f96300f5510 2023-12-21T03:57:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.080+0000 7f962d7fa700 1 --2- 172.21.15.78:0/1255982155 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f962005c2b0 0x7f962005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.080+0000 7f962d7fa700 1 -- 172.21.15.78:0/1255982155 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f96240777a0 con 0x7f96300f5510 2023-12-21T03:57:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.080+0000 7f962ffff700 1 --2- 172.21.15.78:0/1255982155 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f962005c2b0 0x7f962005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.080+0000 7f962ffff700 1 --2- 172.21.15.78:0/1255982155 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f962005c2b0 0x7f962005e770 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f961c004e20 tx=0x7f961c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:21.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.084+0000 7f962d7fa700 1 -- 172.21.15.78:0/1255982155 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f962404a7a0 con 0x7f96300f5510 2023-12-21T03:57:21.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.289+0000 7f9635aa1700 1 -- 172.21.15.78:0/1255982155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f96300fdca0 con 0x7f96300f5510 2023-12-21T03:57:21.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.289+0000 7f962d7fa700 1 -- 172.21.15.78:0/1255982155 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f962404ddc0 con 0x7f96300f5510 2023-12-21T03:57:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:21.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.290+0000 7f9635aa1700 1 -- 172.21.15.78:0/1255982155 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f962005c2b0 msgr2=0x7f962005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:21.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.290+0000 7f9635aa1700 1 --2- 172.21.15.78:0/1255982155 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f962005c2b0 0x7f962005e770 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f961c004e20 tx=0x7f961c00b040).stop 2023-12-21T03:57:21.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.290+0000 7f9635aa1700 1 -- 172.21.15.78:0/1255982155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96300f5510 msgr2=0x7f96300f9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:21.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.290+0000 7f9635aa1700 1 --2- 172.21.15.78:0/1255982155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96300f5510 0x7f96300f9670 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f9624004e30 tx=0x7f9624009b30).stop 2023-12-21T03:57:21.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.291+0000 7f9635aa1700 1 -- 172.21.15.78:0/1255982155 shutdown_connections 2023-12-21T03:57:21.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.291+0000 7f9635aa1700 1 --2- 172.21.15.78:0/1255982155 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f962005c2b0 0x7f962005e770 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:21.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.291+0000 7f9635aa1700 1 --2- 172.21.15.78:0/1255982155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96300f5510 0x7f96300f9670 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:21.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.291+0000 7f9635aa1700 1 --2- 172.21.15.78:0/1255982155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96300f67b0 0x7f96300f7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:21.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.291+0000 7f9635aa1700 1 -- 172.21.15.78:0/1255982155 >> 172.21.15.78:0/1255982155 conn(0x7f96300f0c70 msgr2=0x7f96300603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:21.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.291+0000 7f9635aa1700 1 -- 172.21.15.78:0/1255982155 shutdown_connections 2023-12-21T03:57:21.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:21.291+0000 7f9635aa1700 1 -- 172.21.15.78:0/1255982155 wait complete. 2023-12-21T03:57:21.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:19.698335+0000 mgr.smithi078.zidbgq (mgr.14162) 599 : cluster [DBG] pgmap v560: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:21.566 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:21 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:21.290437+0000 mon.smithi078 (mon.0) 590 : audit [DBG] from='client.? 172.21.15.78:0/1255982155' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:21.658 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:22.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:19.698335+0000 mgr.smithi078.zidbgq (mgr.14162) 599 : cluster [DBG] pgmap v560: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:22.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:21 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:21.290437+0000 mon.smithi078 (mon.0) 590 : audit [DBG] from='client.? 172.21.15.78:0/1255982155' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:23.817 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:23 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:21.698775+0000 mgr.smithi078.zidbgq (mgr.14162) 600 : cluster [DBG] pgmap v561: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:24.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:23 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:21.698775+0000 mgr.smithi078.zidbgq (mgr.14162) 600 : cluster [DBG] pgmap v561: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:24.659 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:24.902 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:25.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:25 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:23.699189+0000 mgr.smithi078.zidbgq (mgr.14162) 601 : cluster [DBG] pgmap v562: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:25.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.925+0000 7f51d5970700 1 -- 172.21.15.78:0/410272200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51d00f7a60 msgr2=0x7f51d00f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:25.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 --2- 172.21.15.78:0/410272200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51d00f7a60 0x7f51d00f7ec0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f51c0004660 tx=0x7f51c0009b30).stop 2023-12-21T03:57:25.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 -- 172.21.15.78:0/410272200 shutdown_connections 2023-12-21T03:57:25.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 --2- 172.21.15.78:0/410272200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51d00f7a60 0x7f51d00f7ec0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:25.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 --2- 172.21.15.78:0/410272200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f51d00f67c0 0x7f51d00f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:57:25.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 -- 172.21.15.78:0/410272200 >> 172.21.15.78:0/410272200 conn(0x7f51d00f1f20 msgr2=0x7f51d00f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:25.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 -- 172.21.15.78:0/410272200 shutdown_connections 2023-12-21T03:57:25.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 -- 172.21.15.78:0/410272200 wait complete. 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 Processor -- start 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 -- start start 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51d00f67c0 0x7f51d01070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f51d00f7a60 0x7f51d01075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d0107be0 con 0x7f51d00f67c0 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.926+0000 7f51d5970700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d0108490 con 0x7f51d00f7a60 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51ceffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51d00f67c0 0x7f51d01070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51ceffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51d00f67c0 0x7f51d01070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49968/0 (socket says 172.21.15.78:49968) 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51ceffd700 1 -- 172.21.15.78:0/767627632 learned_addr learned my addr 172.21.15.78:0/767627632 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51ce7fc700 1 --2- 172.21.15.78:0/767627632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f51d00f7a60 0x7f51d01075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51ceffd700 1 -- 172.21.15.78:0/767627632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f51d00f7a60 msgr2=0x7f51d01075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:25.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51ceffd700 1 --2- 172.21.15.78:0/767627632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f51d00f7a60 0x7f51d01075a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:25.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51ceffd700 1 -- 172.21.15.78:0/767627632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c0005040 con 0x7f51d00f67c0 2023-12-21T03:57:25.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51ceffd700 1 --2- 172.21.15.78:0/767627632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51d00f67c0 0x7f51d01070b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f51c40093f0 tx=0x7f51c4006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:25.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51b77fe700 1 -- 172.21.15.78:0/767627632 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51c4009ea0 con 0x7f51d00f67c0 2023-12-21T03:57:25.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51d5970700 1 -- 172.21.15.78:0/767627632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d0108770 con 0x7f51d00f67c0 2023-12-21T03:57:25.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51b77fe700 1 -- 172.21.15.78:0/767627632 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51c4013af0 con 0x7f51d00f67c0 2023-12-21T03:57:25.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51b77fe700 1 -- 172.21.15.78:0/767627632 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51c400dad0 con 0x7f51d00f67c0 2023-12-21T03:57:25.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.927+0000 7f51d5970700 1 -- 172.21.15.78:0/767627632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d0108d30 con 0x7f51d00f67c0 2023-12-21T03:57:25.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.928+0000 7f51d5970700 1 -- 172.21.15.78:0/767627632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d0059100 con 0x7f51d00f67c0 2023-12-21T03:57:25.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.929+0000 7f51b77fe700 1 -- 172.21.15.78:0/767627632 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f51c401a060 con 0x7f51d00f67c0 2023-12-21T03:57:25.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.929+0000 7f51b77fe700 1 --2- 172.21.15.78:0/767627632 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f51b805c2a0 0x7f51b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:25.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.930+0000 7f51b77fe700 1 -- 172.21.15.78:0/767627632 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f51c4075f90 con 0x7f51d00f67c0 2023-12-21T03:57:25.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.930+0000 7f51ce7fc700 1 --2- 172.21.15.78:0/767627632 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f51b805c2a0 0x7f51b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:25.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.930+0000 7f51ce7fc700 1 --2- 172.21.15.78:0/767627632 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f51b805c2a0 0x7f51b805e760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f51c0013580 tx=0x7f51c0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:25.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:25.933+0000 7f51b77fe700 1 -- 172.21.15.78:0/767627632 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51c4049080 con 0x7f51d00f67c0 2023-12-21T03:57:26.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:25 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:23.699189+0000 mgr.smithi078.zidbgq (mgr.14162) 601 : cluster [DBG] pgmap v562: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:26.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.143+0000 7f51d5970700 1 -- 172.21.15.78:0/767627632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f51d0045ad0 con 0x7f51d00f67c0 2023-12-21T03:57:26.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.143+0000 7f51b77fe700 1 -- 172.21.15.78:0/767627632 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f51c404c6a0 con 0x7f51d00f67c0 2023-12-21T03:57:26.143 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.144+0000 7f51d5970700 1 -- 172.21.15.78:0/767627632 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f51b805c2a0 msgr2=0x7f51b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.144+0000 7f51d5970700 1 --2- 172.21.15.78:0/767627632 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f51b805c2a0 0x7f51b805e760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f51c0013580 tx=0x7f51c0009b30).stop 2023-12-21T03:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.144+0000 7f51d5970700 1 -- 172.21.15.78:0/767627632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51d00f67c0 msgr2=0x7f51d01070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.144+0000 7f51d5970700 1 --2- 172.21.15.78:0/767627632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51d00f67c0 0x7f51d01070b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f51c40093f0 tx=0x7f51c4006ec0).stop 2023-12-21T03:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.145+0000 7f51d5970700 1 -- 172.21.15.78:0/767627632 shutdown_connections 2023-12-21T03:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.145+0000 7f51d5970700 1 --2- 172.21.15.78:0/767627632 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f51b805c2a0 0x7f51b805e760 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.145+0000 7f51d5970700 1 --2- 172.21.15.78:0/767627632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51d00f67c0 0x7f51d01070b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.145+0000 7f51d5970700 1 --2- 172.21.15.78:0/767627632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f51d00f7a60 0x7f51d01075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.145+0000 7f51d5970700 1 -- 172.21.15.78:0/767627632 >> 172.21.15.78:0/767627632 conn(0x7f51d00f1f20 msgr2=0x7f51d00fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.145+0000 7f51d5970700 1 -- 172.21.15.78:0/767627632 shutdown_connections 2023-12-21T03:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:26.145+0000 7f51d5970700 1 -- 172.21.15.78:0/767627632 wait complete. 2023-12-21T03:57:26.524 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:26.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:26 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:26.143406+0000 mon.smithi078 (mon.0) 591 : audit [DBG] from='client.? 172.21.15.78:0/767627632' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:27.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:26 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:26.143406+0000 mon.smithi078 (mon.0) 591 : audit [DBG] from='client.? 172.21.15.78:0/767627632' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:27.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:27 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:25.699488+0000 mgr.smithi078.zidbgq (mgr.14162) 602 : cluster [DBG] pgmap v563: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:28.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:27 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:25.699488+0000 mgr.smithi078.zidbgq (mgr.14162) 602 : cluster [DBG] pgmap v563: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:29.525 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:29.766 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:29.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:29 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:27.699970+0000 mgr.smithi078.zidbgq (mgr.14162) 603 : cluster [DBG] pgmap v564: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:30.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:29 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:27.699970+0000 mgr.smithi078.zidbgq (mgr.14162) 603 : cluster [DBG] pgmap v564: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:30.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.804+0000 7f8cc9626700 1 -- 172.21.15.78:0/963238466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fbc90 msgr2=0x7f8cc40fe080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.804+0000 7f8cc9626700 1 --2- 172.21.15.78:0/963238466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fbc90 0x7f8cc40fe080 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f8cb4004660 tx=0x7f8cb4009b30).stop 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.804+0000 7f8cc9626700 1 -- 172.21.15.78:0/963238466 shutdown_connections 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.804+0000 7f8cc9626700 1 --2- 172.21.15.78:0/963238466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fbc90 0x7f8cc40fe080 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.804+0000 7f8cc9626700 1 --2- 172.21.15.78:0/963238466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cc4060870 0x7f8cc40fb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.804+0000 7f8cc9626700 1 -- 172.21.15.78:0/963238466 >> 172.21.15.78:0/963238466 conn(0x7f8cc40f0e20 msgr2=0x7f8cc40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.804+0000 7f8cc9626700 1 -- 172.21.15.78:0/963238466 shutdown_connections 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.804+0000 7f8cc9626700 1 -- 172.21.15.78:0/963238466 wait complete. 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc9626700 1 Processor -- start 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc9626700 1 -- start start 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc9626700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc4060870 0x7f8cc41092e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc9626700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cc40fbc90 0x7f8cc41097d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc9626700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc4109e10 con 0x7f8cc4060870 2023-12-21T03:57:30.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc9626700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc410a6c0 con 0x7f8cc40fbc90 2023-12-21T03:57:30.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc4060870 0x7f8cc41092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:30.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc4060870 0x7f8cc41092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44590/0 (socket says 172.21.15.78:44590) 2023-12-21T03:57:30.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc2ffd700 1 -- 172.21.15.78:0/2798927683 learned_addr learned my addr 172.21.15.78:0/2798927683 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:30.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc27fc700 1 --2- 172.21.15.78:0/2798927683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cc40fbc90 0x7f8cc41097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:30.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc2ffd700 1 -- 172.21.15.78:0/2798927683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cc40fbc90 msgr2=0x7f8cc41097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:30.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.805+0000 7f8cc2ffd700 1 --2- 172.21.15.78:0/2798927683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cc40fbc90 0x7f8cc41097d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:30.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.806+0000 7f8cc2ffd700 1 -- 172.21.15.78:0/2798927683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb4005040 con 0x7f8cc4060870 2023-12-21T03:57:30.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.806+0000 7f8cc2ffd700 1 --2- 172.21.15.78:0/2798927683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc4060870 0x7f8cc41092e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f8cac00b410 tx=0x7f8cac006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:30.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.806+0000 7f8cbb7fe700 1 -- 172.21.15.78:0/2798927683 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cac0125f0 con 0x7f8cc4060870 2023-12-21T03:57:30.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.806+0000 7f8cbb7fe700 1 -- 172.21.15.78:0/2798927683 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cac012c00 con 0x7f8cc4060870 2023-12-21T03:57:30.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.806+0000 7f8cc9626700 1 -- 172.21.15.78:0/2798927683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc410a9a0 con 0x7f8cc4060870 2023-12-21T03:57:30.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.806+0000 7f8cbb7fe700 1 -- 172.21.15.78:0/2798927683 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cac002aa0 con 0x7f8cc4060870 2023-12-21T03:57:30.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.806+0000 7f8cc9626700 1 -- 172.21.15.78:0/2798927683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc410af60 con 0x7f8cc4060870 2023-12-21T03:57:30.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.807+0000 7f8cc9626700 1 -- 172.21.15.78:0/2798927683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc4059100 con 0x7f8cc4060870 2023-12-21T03:57:30.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.807+0000 7f8cbb7fe700 1 -- 172.21.15.78:0/2798927683 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8cac00f030 con 0x7f8cc4060870 2023-12-21T03:57:30.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.807+0000 7f8cbb7fe700 1 --2- 172.21.15.78:0/2798927683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8cb005c2a0 0x7f8cb005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:30.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.808+0000 7f8cc27fc700 1 --2- 172.21.15.78:0/2798927683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8cb005c2a0 0x7f8cb005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:30.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.808+0000 7f8cbb7fe700 1 -- 172.21.15.78:0/2798927683 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8cac076060 con 0x7f8cc4060870 2023-12-21T03:57:30.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.808+0000 7f8cc27fc700 1 --2- 172.21.15.78:0/2798927683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8cb005c2a0 0x7f8cb005e760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f8cb4004e30 tx=0x7f8cb4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:30.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:30.811+0000 7f8cbb7fe700 1 -- 172.21.15.78:0/2798927683 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cac049000 con 0x7f8cc4060870 2023-12-21T03:57:31.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.017+0000 7f8cc9626700 1 -- 172.21.15.78:0/2798927683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8cc4045ad0 con 0x7f8cc4060870 2023-12-21T03:57:31.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.017+0000 7f8cbb7fe700 1 -- 172.21.15.78:0/2798927683 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f8cac04c620 con 0x7f8cc4060870 2023-12-21T03:57:31.019 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.019+0000 7f8cc9626700 1 -- 172.21.15.78:0/2798927683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8cb005c2a0 msgr2=0x7f8cb005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.019+0000 7f8cc9626700 1 --2- 172.21.15.78:0/2798927683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8cb005c2a0 0x7f8cb005e760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f8cb4004e30 tx=0x7f8cb4009b30).stop 2023-12-21T03:57:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.019+0000 7f8cc9626700 1 -- 172.21.15.78:0/2798927683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc4060870 msgr2=0x7f8cc41092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.019+0000 7f8cc9626700 1 --2- 172.21.15.78:0/2798927683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc4060870 0x7f8cc41092e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f8cac00b410 tx=0x7f8cac006ef0).stop 2023-12-21T03:57:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.019+0000 7f8cc9626700 1 -- 172.21.15.78:0/2798927683 shutdown_connections 2023-12-21T03:57:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.019+0000 7f8cc9626700 1 --2- 172.21.15.78:0/2798927683 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f8cb005c2a0 0x7f8cb005e760 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.019+0000 7f8cc9626700 1 --2- 172.21.15.78:0/2798927683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc4060870 0x7f8cc41092e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.019+0000 7f8cc9626700 1 --2- 172.21.15.78:0/2798927683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cc40fbc90 0x7f8cc41097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.019+0000 7f8cc9626700 1 -- 172.21.15.78:0/2798927683 >> 172.21.15.78:0/2798927683 conn(0x7f8cc40f0e20 msgr2=0x7f8cc40f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.019+0000 7f8cc9626700 1 -- 172.21.15.78:0/2798927683 shutdown_connections 2023-12-21T03:57:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:31.019+0000 7f8cc9626700 1 -- 172.21.15.78:0/2798927683 wait complete. 2023-12-21T03:57:31.448 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:29.700286+0000 mgr.smithi078.zidbgq (mgr.14162) 604 : cluster [DBG] pgmap v565: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:30.574746+0000 mon.smithi078 (mon.0) 592 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:57:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:30.574964+0000 mon.smithi078 (mon.0) 593 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:57:31.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:31 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:31.018915+0000 mon.smithi078 (mon.0) 594 : audit [DBG] from='client.? 172.21.15.78:0/2798927683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:32.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:29.700286+0000 mgr.smithi078.zidbgq (mgr.14162) 604 : cluster [DBG] pgmap v565: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:32.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:30.574746+0000 mon.smithi078 (mon.0) 592 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/trash_purge_schedule"}]: dispatch 2023-12-21T03:57:32.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:30.574964+0000 mon.smithi078 (mon.0) 593 : audit [INF] from='mgr.14162 172.21.15.78:0/3239567821' entity='mgr.smithi078.zidbgq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi078.zidbgq/mirror_snapshot_schedule"}]: dispatch 2023-12-21T03:57:32.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:31 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:31.018915+0000 mon.smithi078 (mon.0) 594 : audit [DBG] from='client.? 172.21.15.78:0/2798927683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:34.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:33 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:31.700731+0000 mgr.smithi078.zidbgq (mgr.14162) 605 : cluster [DBG] pgmap v566: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:34.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:33 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:31.700731+0000 mgr.smithi078.zidbgq (mgr.14162) 605 : cluster [DBG] pgmap v566: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:34.449 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:34.688 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:35.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.682+0000 7fd20633d700 1 -- 172.21.15.78:0/2154576534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd200060870 msgr2=0x7fd2000f93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:35.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.682+0000 7fd20633d700 1 --2- 172.21.15.78:0/2154576534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd200060870 0x7fd2000f93a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fd1f0004660 tx=0x7fd1f0009b30).stop 2023-12-21T03:57:35.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.682+0000 7fd20633d700 1 -- 172.21.15.78:0/2154576534 shutdown_connections 2023-12-21T03:57:35.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.682+0000 7fd20633d700 1 --2- 172.21.15.78:0/2154576534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2000f9920 0x7fd2000fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:35.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.682+0000 7fd20633d700 1 --2- 172.21.15.78:0/2154576534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd200060870 0x7fd2000f93a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:35.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.682+0000 7fd20633d700 1 -- 172.21.15.78:0/2154576534 >> 172.21.15.78:0/2154576534 conn(0x7fd2000f0e20 msgr2=0x7fd2000f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:35.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.682+0000 7fd20633d700 1 -- 172.21.15.78:0/2154576534 shutdown_connections 2023-12-21T03:57:35.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.682+0000 7fd20633d700 1 -- 172.21.15.78:0/2154576534 wait complete. 2023-12-21T03:57:35.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.683+0000 7fd20633d700 1 Processor -- start 2023-12-21T03:57:35.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.683+0000 7fd20633d700 1 -- start start 2023-12-21T03:57:35.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.683+0000 7fd20633d700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd200060870 0x7fd200107040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:35.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.683+0000 7fd20633d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2000f9920 0x7fd200107530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:35.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.683+0000 7fd20633d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd200107ae0 con 0x7fd2000f9920 2023-12-21T03:57:35.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.683+0000 7fd20633d700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd200107c20 con 0x7fd200060870 2023-12-21T03:57:35.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.683+0000 7fd1ff7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2000f9920 0x7fd200107530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:35.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.683+0000 7fd1ff7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2000f9920 0x7fd200107530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44612/0 (socket says 172.21.15.78:44612) 2023-12-21T03:57:35.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.683+0000 7fd1ff7fe700 1 -- 172.21.15.78:0/3411177814 learned_addr learned my addr 172.21.15.78:0/3411177814 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:35.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.683+0000 7fd1fffff700 1 --2- 172.21.15.78:0/3411177814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd200060870 0x7fd200107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:35.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.683+0000 7fd1ff7fe700 1 -- 172.21.15.78:0/3411177814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd200060870 msgr2=0x7fd200107040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:35.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.684+0000 7fd1ff7fe700 1 --2- 172.21.15.78:0/3411177814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd200060870 0x7fd200107040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:35.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.684+0000 7fd1ff7fe700 1 -- 172.21.15.78:0/3411177814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f0005040 con 0x7fd2000f9920 2023-12-21T03:57:35.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.684+0000 7fd1ff7fe700 1 --2- 172.21.15.78:0/3411177814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2000f9920 0x7fd200107530 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fd1f40026d0 tx=0x7fd1f4009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:35.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.684+0000 7fd1fcff9700 1 -- 172.21.15.78:0/3411177814 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1f4009320 con 0x7fd2000f9920 2023-12-21T03:57:35.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.684+0000 7fd1fcff9700 1 -- 172.21.15.78:0/3411177814 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1f400f040 con 0x7fd2000f9920 2023-12-21T03:57:35.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.684+0000 7fd20633d700 1 -- 172.21.15.78:0/3411177814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2001085c0 con 0x7fd2000f9920 2023-12-21T03:57:35.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.684+0000 7fd1fcff9700 1 -- 172.21.15.78:0/3411177814 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1f4014870 con 0x7fd2000f9920 2023-12-21T03:57:35.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.684+0000 7fd20633d700 1 -- 172.21.15.78:0/3411177814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd200108b80 con 0x7fd2000f9920 2023-12-21T03:57:35.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.685+0000 7fd20633d700 1 -- 172.21.15.78:0/3411177814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd200045ad0 con 0x7fd2000f9920 2023-12-21T03:57:35.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.685+0000 7fd1fcff9700 1 -- 172.21.15.78:0/3411177814 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd1f4009480 con 0x7fd2000f9920 2023-12-21T03:57:35.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.685+0000 7fd1fcff9700 1 --2- 172.21.15.78:0/3411177814 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd1e805c2a0 0x7fd1e805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:35.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.685+0000 7fd1fcff9700 1 -- 172.21.15.78:0/3411177814 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd1f4075ed0 con 0x7fd2000f9920 2023-12-21T03:57:35.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.685+0000 7fd1fffff700 1 --2- 172.21.15.78:0/3411177814 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd1e805c2a0 0x7fd1e805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:35.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.686+0000 7fd1fffff700 1 --2- 172.21.15.78:0/3411177814 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd1e805c2a0 0x7fd1e805e760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fd1f0004e30 tx=0x7fd1f0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:35.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.689+0000 7fd1fcff9700 1 -- 172.21.15.78:0/3411177814 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1f4048ed0 con 0x7fd2000f9920 2023-12-21T03:57:35.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:35 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:33.701216+0000 mgr.smithi078.zidbgq (mgr.14162) 606 : cluster [DBG] pgmap v567: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:35.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.897+0000 7fd20633d700 1 -- 172.21.15.78:0/3411177814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd200027a00 con 0x7fd2000f9920 2023-12-21T03:57:35.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.897+0000 7fd1fcff9700 1 -- 172.21.15.78:0/3411177814 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fd1f404c4f0 con 0x7fd2000f9920 2023-12-21T03:57:35.898 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.899+0000 7fd20633d700 1 -- 172.21.15.78:0/3411177814 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd1e805c2a0 msgr2=0x7fd1e805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.899+0000 7fd20633d700 1 --2- 172.21.15.78:0/3411177814 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd1e805c2a0 0x7fd1e805e760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fd1f0004e30 tx=0x7fd1f0009b30).stop 2023-12-21T03:57:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.899+0000 7fd20633d700 1 -- 172.21.15.78:0/3411177814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2000f9920 msgr2=0x7fd200107530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.899+0000 7fd20633d700 1 --2- 172.21.15.78:0/3411177814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2000f9920 0x7fd200107530 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fd1f40026d0 tx=0x7fd1f4009e00).stop 2023-12-21T03:57:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.899+0000 7fd20633d700 1 -- 172.21.15.78:0/3411177814 shutdown_connections 2023-12-21T03:57:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.899+0000 7fd20633d700 1 --2- 172.21.15.78:0/3411177814 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fd1e805c2a0 0x7fd1e805e760 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.899+0000 7fd20633d700 1 --2- 172.21.15.78:0/3411177814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd200060870 0x7fd200107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.899+0000 7fd20633d700 1 --2- 172.21.15.78:0/3411177814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2000f9920 0x7fd200107530 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.899+0000 7fd20633d700 1 -- 172.21.15.78:0/3411177814 >> 172.21.15.78:0/3411177814 conn(0x7fd2000f0e20 msgr2=0x7fd2000f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.899+0000 7fd20633d700 1 -- 172.21.15.78:0/3411177814 shutdown_connections 2023-12-21T03:57:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:35.899+0000 7fd20633d700 1 -- 172.21.15.78:0/3411177814 wait complete. 2023-12-21T03:57:36.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:35 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:33.701216+0000 mgr.smithi078.zidbgq (mgr.14162) 606 : cluster [DBG] pgmap v567: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:36.416 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:36.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:36 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:35.898524+0000 mon.smithi078 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.78:0/3411177814' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:37.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:36 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:35.898524+0000 mon.smithi078 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.78:0/3411177814' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:38.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:37 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:35.701544+0000 mgr.smithi078.zidbgq (mgr.14162) 607 : cluster [DBG] pgmap v568: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:38.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:37 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:35.701544+0000 mgr.smithi078.zidbgq (mgr.14162) 607 : cluster [DBG] pgmap v568: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:39.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:39.784 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:39 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:37.702040+0000 mgr.smithi078.zidbgq (mgr.14162) 608 : cluster [DBG] pgmap v569: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:40.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:39 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:37.702040+0000 mgr.smithi078.zidbgq (mgr.14162) 608 : cluster [DBG] pgmap v569: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.608+0000 7f8805bb5700 1 -- 172.21.15.78:0/2488478976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88000f6960 msgr2=0x7f88000f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.608+0000 7f8805bb5700 1 --2- 172.21.15.78:0/2488478976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88000f6960 0x7f88000f6dc0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f87f4004660 tx=0x7f87f4009b30).stop 2023-12-21T03:57:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.608+0000 7f8805bb5700 1 -- 172.21.15.78:0/2488478976 shutdown_connections 2023-12-21T03:57:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.608+0000 7f8805bb5700 1 --2- 172.21.15.78:0/2488478976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88000f6960 0x7f88000f6dc0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.608+0000 7f8805bb5700 1 --2- 172.21.15.78:0/2488478976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88000f56c0 0x7f88000f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.608+0000 7f8805bb5700 1 -- 172.21.15.78:0/2488478976 >> 172.21.15.78:0/2488478976 conn(0x7f88000f0e20 msgr2=0x7f88000f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.608+0000 7f8805bb5700 1 -- 172.21.15.78:0/2488478976 shutdown_connections 2023-12-21T03:57:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.608+0000 7f8805bb5700 1 -- 172.21.15.78:0/2488478976 wait complete. 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f8805bb5700 1 Processor -- start 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f8805bb5700 1 -- start start 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f8805bb5700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88000f56c0 0x7f88001070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f8805bb5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88000f6960 0x7f88001075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f8805bb5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8800107b70 con 0x7f88000f6960 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f8805bb5700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8800107cb0 con 0x7f88000f56c0 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f87feffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88000f6960 0x7f88001075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f87feffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88000f6960 0x7f88001075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54854/0 (socket says 172.21.15.78:54854) 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f87feffd700 1 -- 172.21.15.78:0/2510903158 learned_addr learned my addr 172.21.15.78:0/2510903158 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f87ff7fe700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88000f56c0 0x7f88001070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f87feffd700 1 -- 172.21.15.78:0/2510903158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88000f56c0 msgr2=0x7f88001070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f87feffd700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88000f56c0 0x7f88001070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f87feffd700 1 -- 172.21.15.78:0/2510903158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4005040 con 0x7f88000f6960 2023-12-21T03:57:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f87ff7fe700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88000f56c0 0x7f88001070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:57:40.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.609+0000 7f87feffd700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88000f6960 0x7f88001075c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f87f4009830 tx=0x7f87f4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:40.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.610+0000 7f87e7fff700 1 -- 172.21.15.78:0/2510903158 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87f4018070 con 0x7f88000f6960 2023-12-21T03:57:40.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.610+0000 7f8805bb5700 1 -- 172.21.15.78:0/2510903158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88001085f0 con 0x7f88000f6960 2023-12-21T03:57:40.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.610+0000 7f87e7fff700 1 -- 172.21.15.78:0/2510903158 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87f4013410 con 0x7f88000f6960 2023-12-21T03:57:40.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.610+0000 7f87e7fff700 1 -- 172.21.15.78:0/2510903158 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87f401c620 con 0x7f88000f6960 2023-12-21T03:57:40.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.610+0000 7f8805bb5700 1 -- 172.21.15.78:0/2510903158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8800108b50 con 0x7f88000f6960 2023-12-21T03:57:40.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.611+0000 7f8805bb5700 1 -- 172.21.15.78:0/2510903158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8800045ad0 con 0x7f88000f6960 2023-12-21T03:57:40.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.611+0000 7f87e7fff700 1 -- 172.21.15.78:0/2510903158 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f87f401c8e0 con 0x7f88000f6960 2023-12-21T03:57:40.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.612+0000 7f87e7fff700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f87e805c2a0 0x7f87e805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:40.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.612+0000 7f87e7fff700 1 -- 172.21.15.78:0/2510903158 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f87f4077a60 con 0x7f88000f6960 2023-12-21T03:57:40.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.612+0000 7f87ff7fe700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f87e805c2a0 0x7f87e805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:40.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.612+0000 7f87ff7fe700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f87e805c2a0 0x7f87e805e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f88000f67c0 tx=0x7f87f0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.616+0000 7f87e7fff700 1 -- 172.21.15.78:0/2510903158 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87f404aa00 con 0x7f88000f6960 2023-12-21T03:57:40.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.821+0000 7f8805bb5700 1 -- 172.21.15.78:0/2510903158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8800109250 con 0x7f88000f6960 2023-12-21T03:57:40.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.822+0000 7f87e7fff700 1 -- 172.21.15.78:0/2510903158 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f87f404e020 con 0x7f88000f6960 2023-12-21T03:57:40.823 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:40.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.823+0000 7f8805bb5700 1 -- 172.21.15.78:0/2510903158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f87e805c2a0 msgr2=0x7f87e805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:40.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.823+0000 7f8805bb5700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f87e805c2a0 0x7f87e805e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f88000f67c0 tx=0x7f87f0009040).stop 2023-12-21T03:57:40.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.823+0000 7f8805bb5700 1 -- 172.21.15.78:0/2510903158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88000f6960 msgr2=0x7f88001075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:40.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.823+0000 7f8805bb5700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88000f6960 0x7f88001075c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f87f4009830 tx=0x7f87f4009b30).stop 2023-12-21T03:57:40.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.823+0000 7f8805bb5700 1 -- 172.21.15.78:0/2510903158 shutdown_connections 2023-12-21T03:57:40.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.823+0000 7f8805bb5700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f87e805c2a0 0x7f87e805e760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:40.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.823+0000 7f8805bb5700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88000f56c0 0x7f88001070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:40.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.823+0000 7f8805bb5700 1 --2- 172.21.15.78:0/2510903158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88000f6960 0x7f88001075c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:40.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.823+0000 7f8805bb5700 1 -- 172.21.15.78:0/2510903158 >> 172.21.15.78:0/2510903158 conn(0x7f88000f0e20 msgr2=0x7f88000603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:40.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.823+0000 7f8805bb5700 1 -- 172.21.15.78:0/2510903158 shutdown_connections 2023-12-21T03:57:40.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:40.824+0000 7f8805bb5700 1 -- 172.21.15.78:0/2510903158 wait complete. 2023-12-21T03:57:41.188 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:42.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:39.702327+0000 mgr.smithi078.zidbgq (mgr.14162) 609 : cluster [DBG] pgmap v570: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:42.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:41 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:40.823161+0000 mon.smithi078 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.78:0/2510903158' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:42.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:39.702327+0000 mgr.smithi078.zidbgq (mgr.14162) 609 : cluster [DBG] pgmap v570: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:42.067 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:41 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:40.823161+0000 mon.smithi078 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.78:0/2510903158' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:44.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:43 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:41.702604+0000 mgr.smithi078.zidbgq (mgr.14162) 610 : cluster [DBG] pgmap v571: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:44.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:43 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:41.702604+0000 mgr.smithi078.zidbgq (mgr.14162) 610 : cluster [DBG] pgmap v571: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:44.189 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:44.425 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:45.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.368+0000 7f96d1f3c700 1 -- 172.21.15.78:0/3760844443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96cc0f99a0 msgr2=0x7f96cc0fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.368+0000 7f96d1f3c700 1 --2- 172.21.15.78:0/3760844443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96cc0f99a0 0x7f96cc0fbd90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f96b4004660 tx=0x7f96b4009b30).stop 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.368+0000 7f96d1f3c700 1 -- 172.21.15.78:0/3760844443 shutdown_connections 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.368+0000 7f96d1f3c700 1 --2- 172.21.15.78:0/3760844443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96cc0f99a0 0x7f96cc0fbd90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.368+0000 7f96d1f3c700 1 --2- 172.21.15.78:0/3760844443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96cc060810 0x7f96cc0f9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.368+0000 7f96d1f3c700 1 -- 172.21.15.78:0/3760844443 >> 172.21.15.78:0/3760844443 conn(0x7f96cc0f0e20 msgr2=0x7f96cc0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.368+0000 7f96d1f3c700 1 -- 172.21.15.78:0/3760844443 shutdown_connections 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.368+0000 7f96d1f3c700 1 -- 172.21.15.78:0/3760844443 wait complete. 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.369+0000 7f96d1f3c700 1 Processor -- start 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.369+0000 7f96d1f3c700 1 -- start start 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.369+0000 7f96d1f3c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96cc060810 0x7f96cc107090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.369+0000 7f96d1f3c700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96cc0f99a0 0x7f96cc107580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.369+0000 7f96d1f3c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc107b30 con 0x7f96cc060810 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.369+0000 7f96d1f3c700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc107c70 con 0x7f96cc0f99a0 2023-12-21T03:57:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.369+0000 7f96cb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96cc060810 0x7f96cc107090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.369+0000 7f96cb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96cc060810 0x7f96cc107090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54878/0 (socket says 172.21.15.78:54878) 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.369+0000 7f96cb7fe700 1 -- 172.21.15.78:0/1646597542 learned_addr learned my addr 172.21.15.78:0/1646597542 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.370+0000 7f96caffd700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96cc0f99a0 0x7f96cc107580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.370+0000 7f96cb7fe700 1 -- 172.21.15.78:0/1646597542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96cc0f99a0 msgr2=0x7f96cc107580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.370+0000 7f96cb7fe700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96cc0f99a0 0x7f96cc107580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.370+0000 7f96cb7fe700 1 -- 172.21.15.78:0/1646597542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b4005040 con 0x7f96cc060810 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.370+0000 7f96caffd700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96cc0f99a0 0x7f96cc107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.370+0000 7f96cb7fe700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96cc060810 0x7f96cc107090 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f96bc00c530 tx=0x7f96bc007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.370+0000 7f96c3fff700 1 -- 172.21.15.78:0/1646597542 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96bc00f650 con 0x7f96cc060810 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.370+0000 7f96d1f3c700 1 -- 172.21.15.78:0/1646597542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc108610 con 0x7f96cc060810 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.370+0000 7f96c3fff700 1 -- 172.21.15.78:0/1646597542 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96bc00fc60 con 0x7f96cc060810 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.370+0000 7f96c3fff700 1 -- 172.21.15.78:0/1646597542 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96bc012c80 con 0x7f96cc060810 2023-12-21T03:57:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.370+0000 7f96d1f3c700 1 -- 172.21.15.78:0/1646597542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc108bd0 con 0x7f96cc060810 2023-12-21T03:57:45.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.371+0000 7f96d1f3c700 1 -- 172.21.15.78:0/1646597542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96cc045ad0 con 0x7f96cc060810 2023-12-21T03:57:45.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.371+0000 7f96c3fff700 1 -- 172.21.15.78:0/1646597542 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f96bc010070 con 0x7f96cc060810 2023-12-21T03:57:45.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.371+0000 7f96c3fff700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f96b805c2a0 0x7f96b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:45.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.371+0000 7f96c3fff700 1 -- 172.21.15.78:0/1646597542 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f96bc0758c0 con 0x7f96cc060810 2023-12-21T03:57:45.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.372+0000 7f96caffd700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f96b805c2a0 0x7f96b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:45.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.372+0000 7f96caffd700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f96b805c2a0 0x7f96b805e760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f96b4004e30 tx=0x7f96b4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:45.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.375+0000 7f96c3fff700 1 -- 172.21.15.78:0/1646597542 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96bc0452b0 con 0x7f96cc060810 2023-12-21T03:57:45.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.584+0000 7f96d1f3c700 1 -- 172.21.15.78:0/1646597542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f96cc0ff0f0 con 0x7f96cc060810 2023-12-21T03:57:45.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.584+0000 7f96c3fff700 1 -- 172.21.15.78:0/1646597542 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f96bc0488c0 con 0x7f96cc060810 2023-12-21T03:57:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:45.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.586+0000 7f96d1f3c700 1 -- 172.21.15.78:0/1646597542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f96b805c2a0 msgr2=0x7f96b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:45.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.586+0000 7f96d1f3c700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f96b805c2a0 0x7f96b805e760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f96b4004e30 tx=0x7f96b4009b30).stop 2023-12-21T03:57:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.586+0000 7f96d1f3c700 1 -- 172.21.15.78:0/1646597542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96cc060810 msgr2=0x7f96cc107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.586+0000 7f96d1f3c700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96cc060810 0x7f96cc107090 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f96bc00c530 tx=0x7f96bc007e20).stop 2023-12-21T03:57:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.586+0000 7f96d1f3c700 1 -- 172.21.15.78:0/1646597542 shutdown_connections 2023-12-21T03:57:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.586+0000 7f96d1f3c700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f96b805c2a0 0x7f96b805e760 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.586+0000 7f96d1f3c700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96cc060810 0x7f96cc107090 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.586+0000 7f96d1f3c700 1 --2- 172.21.15.78:0/1646597542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96cc0f99a0 0x7f96cc107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.586+0000 7f96d1f3c700 1 -- 172.21.15.78:0/1646597542 >> 172.21.15.78:0/1646597542 conn(0x7f96cc0f0e20 msgr2=0x7f96cc0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.586+0000 7f96d1f3c700 1 -- 172.21.15.78:0/1646597542 shutdown_connections 2023-12-21T03:57:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:45.586+0000 7f96d1f3c700 1 -- 172.21.15.78:0/1646597542 wait complete. 2023-12-21T03:57:45.971 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:43.702879+0000 mgr.smithi078.zidbgq (mgr.14162) 611 : cluster [DBG] pgmap v572: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:45.971 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:45 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:45.585777+0000 mon.smithi078 (mon.0) 597 : audit [DBG] from='client.? 172.21.15.78:0/1646597542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:45.991 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:46.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:43.702879+0000 mgr.smithi078.zidbgq (mgr.14162) 611 : cluster [DBG] pgmap v572: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:46.016 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:45 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:45.585777+0000 mon.smithi078 (mon.0) 597 : audit [DBG] from='client.? 172.21.15.78:0/1646597542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:48.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:47 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:45.703170+0000 mgr.smithi078.zidbgq (mgr.14162) 612 : cluster [DBG] pgmap v573: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:48.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:47 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:45.703170+0000 mgr.smithi078.zidbgq (mgr.14162) 612 : cluster [DBG] pgmap v573: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:48.993 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:49.233 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:49.954 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:49 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:47.703522+0000 mgr.smithi078.zidbgq (mgr.14162) 613 : cluster [DBG] pgmap v574: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:50.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:49 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:47.703522+0000 mgr.smithi078.zidbgq (mgr.14162) 613 : cluster [DBG] pgmap v574: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.272+0000 7f6556ebb700 1 -- 172.21.15.78:0/3119482757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65500f56c0 msgr2=0x7f65500f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.272+0000 7f6556ebb700 1 --2- 172.21.15.78:0/3119482757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65500f56c0 0x7f65500f5ae0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f65480099e0 tx=0x7f6548004d10).stop 2023-12-21T03:57:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.272+0000 7f6556ebb700 1 -- 172.21.15.78:0/3119482757 shutdown_connections 2023-12-21T03:57:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.272+0000 7f6556ebb700 1 --2- 172.21.15.78:0/3119482757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65500f6960 0x7f65500f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.272+0000 7f6556ebb700 1 --2- 172.21.15.78:0/3119482757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65500f56c0 0x7f65500f5ae0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.272+0000 7f6556ebb700 1 -- 172.21.15.78:0/3119482757 >> 172.21.15.78:0/3119482757 conn(0x7f65500f0e20 msgr2=0x7f65500f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.272+0000 7f6556ebb700 1 -- 172.21.15.78:0/3119482757 shutdown_connections 2023-12-21T03:57:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.272+0000 7f6556ebb700 1 -- 172.21.15.78:0/3119482757 wait complete. 2023-12-21T03:57:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.272+0000 7f6556ebb700 1 Processor -- start 2023-12-21T03:57:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.272+0000 7f6556ebb700 1 -- start start 2023-12-21T03:57:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f6556ebb700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65500f56c0 0x7f65500fa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f6556ebb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65500f6960 0x7f65500f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f6556ebb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65500f8a20 con 0x7f65500f6960 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f6556ebb700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65500f8b60 con 0x7f65500f56c0 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f654ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65500f6960 0x7f65500f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f654ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65500f6960 0x7f65500f84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47214/0 (socket says 172.21.15.78:47214) 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f654ffff700 1 -- 172.21.15.78:0/2948875164 learned_addr learned my addr 172.21.15.78:0/2948875164 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f6554c57700 1 --2- 172.21.15.78:0/2948875164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65500f56c0 0x7f65500fa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f654ffff700 1 -- 172.21.15.78:0/2948875164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65500f56c0 msgr2=0x7f65500fa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f654ffff700 1 --2- 172.21.15.78:0/2948875164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65500f56c0 0x7f65500fa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f654ffff700 1 -- 172.21.15.78:0/2948875164 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6548009640 con 0x7f65500f6960 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f654ffff700 1 --2- 172.21.15.78:0/2948875164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65500f6960 0x7f65500f84c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f653800c530 tx=0x7f6538007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f654d7fa700 1 -- 172.21.15.78:0/2948875164 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f653800f650 con 0x7f65500f6960 2023-12-21T03:57:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f6556ebb700 1 -- 172.21.15.78:0/2948875164 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65500f8e40 con 0x7f65500f6960 2023-12-21T03:57:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f654d7fa700 1 -- 172.21.15.78:0/2948875164 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f653800fc60 con 0x7f65500f6960 2023-12-21T03:57:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.273+0000 7f654d7fa700 1 -- 172.21.15.78:0/2948875164 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6538012c80 con 0x7f65500f6960 2023-12-21T03:57:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.274+0000 7f6556ebb700 1 -- 172.21.15.78:0/2948875164 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655010f200 con 0x7f65500f6960 2023-12-21T03:57:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.274+0000 7f6556ebb700 1 -- 172.21.15.78:0/2948875164 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6550059100 con 0x7f65500f6960 2023-12-21T03:57:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.275+0000 7f654d7fa700 1 -- 172.21.15.78:0/2948875164 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6538010070 con 0x7f65500f6960 2023-12-21T03:57:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.275+0000 7f654d7fa700 1 --2- 172.21.15.78:0/2948875164 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f653c05c2a0 0x7f653c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:50.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.275+0000 7f654d7fa700 1 -- 172.21.15.78:0/2948875164 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f65380758c0 con 0x7f65500f6960 2023-12-21T03:57:50.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.275+0000 7f6554c57700 1 --2- 172.21.15.78:0/2948875164 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f653c05c2a0 0x7f653c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:50.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.276+0000 7f6554c57700 1 --2- 172.21.15.78:0/2948875164 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f653c05c2a0 0x7f653c05e760 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f654800b620 tx=0x7f6548004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.279+0000 7f654d7fa700 1 -- 172.21.15.78:0/2948875164 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65380452b0 con 0x7f65500f6960 2023-12-21T03:57:50.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.485+0000 7f6556ebb700 1 -- 172.21.15.78:0/2948875164 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6550045ad0 con 0x7f65500f6960 2023-12-21T03:57:50.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.485+0000 7f654d7fa700 1 -- 172.21.15.78:0/2948875164 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f653804c2f0 con 0x7f65500f6960 2023-12-21T03:57:50.487 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:50.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.487+0000 7f6556ebb700 1 -- 172.21.15.78:0/2948875164 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f653c05c2a0 msgr2=0x7f653c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:50.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.487+0000 7f6556ebb700 1 --2- 172.21.15.78:0/2948875164 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f653c05c2a0 0x7f653c05e760 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f654800b620 tx=0x7f6548004d10).stop 2023-12-21T03:57:50.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.487+0000 7f6556ebb700 1 -- 172.21.15.78:0/2948875164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65500f6960 msgr2=0x7f65500f84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:50.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.487+0000 7f6556ebb700 1 --2- 172.21.15.78:0/2948875164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65500f6960 0x7f65500f84c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f653800c530 tx=0x7f6538007e20).stop 2023-12-21T03:57:50.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.487+0000 7f6556ebb700 1 -- 172.21.15.78:0/2948875164 shutdown_connections 2023-12-21T03:57:50.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.487+0000 7f6556ebb700 1 --2- 172.21.15.78:0/2948875164 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f653c05c2a0 0x7f653c05e760 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:50.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.487+0000 7f6556ebb700 1 --2- 172.21.15.78:0/2948875164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65500f56c0 0x7f65500fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:50.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.487+0000 7f6556ebb700 1 --2- 172.21.15.78:0/2948875164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65500f6960 0x7f65500f84c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:50.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.487+0000 7f6556ebb700 1 -- 172.21.15.78:0/2948875164 >> 172.21.15.78:0/2948875164 conn(0x7f65500f0e20 msgr2=0x7f655005fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:50.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.487+0000 7f6556ebb700 1 -- 172.21.15.78:0/2948875164 shutdown_connections 2023-12-21T03:57:50.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:50.487+0000 7f6556ebb700 1 -- 172.21.15.78:0/2948875164 wait complete. 2023-12-21T03:57:50.816 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:50 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:50.486751+0000 mon.smithi078 (mon.0) 598 : audit [DBG] from='client.? 172.21.15.78:0/2948875164' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:50.863 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:51.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:50 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:50.486751+0000 mon.smithi078 (mon.0) 598 : audit [DBG] from='client.? 172.21.15.78:0/2948875164' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:52.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:51 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:49.703809+0000 mgr.smithi078.zidbgq (mgr.14162) 614 : cluster [DBG] pgmap v575: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:52.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:51 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:49.703809+0000 mgr.smithi078.zidbgq (mgr.14162) 614 : cluster [DBG] pgmap v575: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:53.864 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:53.897 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:53 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:51.704068+0000 mgr.smithi078.zidbgq (mgr.14162) 615 : cluster [DBG] pgmap v576: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:54.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:53 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:51.704068+0000 mgr.smithi078.zidbgq (mgr.14162) 615 : cluster [DBG] pgmap v576: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:54.097 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.104+0000 7f242775c700 1 -- 172.21.15.78:0/651344199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24200f67e0 msgr2=0x7f24200f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.104+0000 7f242775c700 1 --2- 172.21.15.78:0/651344199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24200f67e0 0x7f24200f6c00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f241c004660 tx=0x7f241c009b30).stop 2023-12-21T03:57:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.104+0000 7f242775c700 1 -- 172.21.15.78:0/651344199 shutdown_connections 2023-12-21T03:57:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.104+0000 7f242775c700 1 --2- 172.21.15.78:0/651344199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f24200f7a80 0x7f24200f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.104+0000 7f242775c700 1 --2- 172.21.15.78:0/651344199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24200f67e0 0x7f24200f6c00 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.104+0000 7f242775c700 1 -- 172.21.15.78:0/651344199 >> 172.21.15.78:0/651344199 conn(0x7f24200f1f20 msgr2=0x7f24200f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.104+0000 7f242775c700 1 -- 172.21.15.78:0/651344199 shutdown_connections 2023-12-21T03:57:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.104+0000 7f242775c700 1 -- 172.21.15.78:0/651344199 wait complete. 2023-12-21T03:57:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f242775c700 1 Processor -- start 2023-12-21T03:57:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f242775c700 1 -- start start 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f242775c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24200f67e0 0x7f242006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f242775c700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f24200f7a80 0x7f2420069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f242775c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242006a380 con 0x7f24200f67e0 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f242775c700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242006a4c0 con 0x7f24200f7a80 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f24254f8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24200f67e0 0x7f242006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f24254f8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24200f67e0 0x7f242006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47228/0 (socket says 172.21.15.78:47228) 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f24254f8700 1 -- 172.21.15.78:0/1244892217 learned_addr learned my addr 172.21.15.78:0/1244892217 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f2424cf7700 1 --2- 172.21.15.78:0/1244892217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f24200f7a80 0x7f2420069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f24254f8700 1 -- 172.21.15.78:0/1244892217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f24200f7a80 msgr2=0x7f2420069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f24254f8700 1 --2- 172.21.15.78:0/1244892217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f24200f7a80 0x7f2420069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f24254f8700 1 -- 172.21.15.78:0/1244892217 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c005040 con 0x7f24200f67e0 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.105+0000 7f24254f8700 1 --2- 172.21.15.78:0/1244892217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24200f67e0 0x7f242006b9a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f241c004e30 tx=0x7f241c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.106+0000 7f2415ffb700 1 -- 172.21.15.78:0/1244892217 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f241c018070 con 0x7f24200f67e0 2023-12-21T03:57:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.106+0000 7f242775c700 1 -- 172.21.15.78:0/1244892217 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242006a740 con 0x7f24200f67e0 2023-12-21T03:57:55.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.106+0000 7f2415ffb700 1 -- 172.21.15.78:0/1244892217 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f241c013410 con 0x7f24200f67e0 2023-12-21T03:57:55.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.106+0000 7f2415ffb700 1 -- 172.21.15.78:0/1244892217 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f241c01c3c0 con 0x7f24200f67e0 2023-12-21T03:57:55.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.106+0000 7f242775c700 1 -- 172.21.15.78:0/1244892217 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242006aca0 con 0x7f24200f67e0 2023-12-21T03:57:55.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.107+0000 7f242775c700 1 -- 172.21.15.78:0/1244892217 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2420045ad0 con 0x7f24200f67e0 2023-12-21T03:57:55.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.107+0000 7f2415ffb700 1 -- 172.21.15.78:0/1244892217 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f241c01d420 con 0x7f24200f67e0 2023-12-21T03:57:55.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.107+0000 7f2415ffb700 1 --2- 172.21.15.78:0/1244892217 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f240c05c2a0 0x7f240c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:55.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.107+0000 7f2415ffb700 1 -- 172.21.15.78:0/1244892217 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f241c077820 con 0x7f24200f67e0 2023-12-21T03:57:55.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.107+0000 7f2424cf7700 1 --2- 172.21.15.78:0/1244892217 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f240c05c2a0 0x7f240c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:55.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.108+0000 7f2424cf7700 1 --2- 172.21.15.78:0/1244892217 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f240c05c2a0 0x7f240c05e760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f2410009880 tx=0x7f241000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:55.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.111+0000 7f2415ffb700 1 -- 172.21.15.78:0/1244892217 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f241c04a820 con 0x7f24200f67e0 2023-12-21T03:57:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.317+0000 7f242775c700 1 -- 172.21.15.78:0/1244892217 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2420027a00 con 0x7f24200f67e0 2023-12-21T03:57:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.318+0000 7f2415ffb700 1 -- 172.21.15.78:0/1244892217 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f241c04de40 con 0x7f24200f67e0 2023-12-21T03:57:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:57:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.319+0000 7f242775c700 1 -- 172.21.15.78:0/1244892217 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f240c05c2a0 msgr2=0x7f240c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.319+0000 7f242775c700 1 --2- 172.21.15.78:0/1244892217 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f240c05c2a0 0x7f240c05e760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f2410009880 tx=0x7f241000b040).stop 2023-12-21T03:57:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.319+0000 7f242775c700 1 -- 172.21.15.78:0/1244892217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24200f67e0 msgr2=0x7f242006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.319+0000 7f242775c700 1 --2- 172.21.15.78:0/1244892217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24200f67e0 0x7f242006b9a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f241c004e30 tx=0x7f241c009b30).stop 2023-12-21T03:57:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.319+0000 7f242775c700 1 -- 172.21.15.78:0/1244892217 shutdown_connections 2023-12-21T03:57:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.319+0000 7f242775c700 1 --2- 172.21.15.78:0/1244892217 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7f240c05c2a0 0x7f240c05e760 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.319+0000 7f242775c700 1 --2- 172.21.15.78:0/1244892217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24200f67e0 0x7f242006b9a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.319+0000 7f242775c700 1 --2- 172.21.15.78:0/1244892217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f24200f7a80 0x7f2420069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.319+0000 7f242775c700 1 -- 172.21.15.78:0/1244892217 >> 172.21.15.78:0/1244892217 conn(0x7f24200f1f20 msgr2=0x7f24200fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.319+0000 7f242775c700 1 -- 172.21.15.78:0/1244892217 shutdown_connections 2023-12-21T03:57:55.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:55.319+0000 7f242775c700 1 -- 172.21.15.78:0/1244892217 wait complete. 2023-12-21T03:57:55.719 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:57:56.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:53.704388+0000 mgr.smithi078.zidbgq (mgr.14162) 616 : cluster [DBG] pgmap v577: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:56.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:55 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:57:55.319140+0000 mon.smithi078 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.78:0/1244892217' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:56.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:53.704388+0000 mgr.smithi078.zidbgq (mgr.14162) 616 : cluster [DBG] pgmap v577: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:56.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:55 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:57:55.319140+0000 mon.smithi078 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.78:0/1244892217' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:57:58.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:57 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:55.704699+0000 mgr.smithi078.zidbgq (mgr.14162) 617 : cluster [DBG] pgmap v578: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:58.066 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:57 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:55.704699+0000 mgr.smithi078.zidbgq (mgr.14162) 617 : cluster [DBG] pgmap v578: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:58.720 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-21T03:57:58.955 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/mon.smithi078/config 2023-12-21T03:57:59.015 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:57:58 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: cluster 2023-12-21T03:57:57.705219+0000 mgr.smithi078.zidbgq (mgr.14162) 618 : cluster [DBG] pgmap v579: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:59.057 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:57:58 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: cluster 2023-12-21T03:57:57.705219+0000 mgr.smithi078.zidbgq (mgr.14162) 618 : cluster [DBG] pgmap v579: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-12-21T03:57:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.952+0000 7fa07dd87700 1 -- 172.21.15.78:0/449400712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0780f67c0 msgr2=0x7fa0780f6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.952+0000 7fa07dd87700 1 --2- 172.21.15.78:0/449400712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0780f67c0 0x7fa0780f6c20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fa064004660 tx=0x7fa064009b30).stop 2023-12-21T03:57:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.952+0000 7fa07dd87700 1 -- 172.21.15.78:0/449400712 shutdown_connections 2023-12-21T03:57:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.952+0000 7fa07dd87700 1 --2- 172.21.15.78:0/449400712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0780f67c0 0x7fa0780f6c20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.952+0000 7fa07dd87700 1 --2- 172.21.15.78:0/449400712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0780f5520 0x7fa0780f5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.952+0000 7fa07dd87700 1 -- 172.21.15.78:0/449400712 >> 172.21.15.78:0/449400712 conn(0x7fa0780f0ca0 msgr2=0x7fa0780f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:57:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.952+0000 7fa07dd87700 1 -- 172.21.15.78:0/449400712 shutdown_connections 2023-12-21T03:57:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.952+0000 7fa07dd87700 1 -- 172.21.15.78:0/449400712 wait complete. 2023-12-21T03:57:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.953+0000 7fa07dd87700 1 Processor -- start 2023-12-21T03:57:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.953+0000 7fa07dd87700 1 -- start start 2023-12-21T03:57:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.953+0000 7fa07dd87700 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0780f5520 0x7fa0781090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.953+0000 7fa07dd87700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0780f67c0 0x7fa0781095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.953+0000 7fa07dd87700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa078109b60 con 0x7fa0780f67c0 2023-12-21T03:57:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.953+0000 7fa07dd87700 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa078109ca0 con 0x7fa0780f5520 2023-12-21T03:57:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.953+0000 7fa077fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0780f67c0 0x7fa0781095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.953+0000 7fa077fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0780f67c0 0x7fa0781095b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46478/0 (socket says 172.21.15.78:46478) 2023-12-21T03:57:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.953+0000 7fa077fff700 1 -- 172.21.15.78:0/2067099361 learned_addr learned my addr 172.21.15.78:0/2067099361 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.954+0000 7fa07cd85700 1 --2- 172.21.15.78:0/2067099361 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0780f5520 0x7fa0781090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.954+0000 7fa077fff700 1 -- 172.21.15.78:0/2067099361 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0780f5520 msgr2=0x7fa0781090c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.954+0000 7fa077fff700 1 --2- 172.21.15.78:0/2067099361 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0780f5520 0x7fa0781090c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.954+0000 7fa077fff700 1 -- 172.21.15.78:0/2067099361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa064005040 con 0x7fa0780f67c0 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.954+0000 7fa077fff700 1 --2- 172.21.15.78:0/2067099361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0780f67c0 0x7fa0781095b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fa064009830 tx=0x7fa064009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.954+0000 7fa0757fa700 1 -- 172.21.15.78:0/2067099361 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa064018070 con 0x7fa0780f67c0 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.954+0000 7fa07dd87700 1 -- 172.21.15.78:0/2067099361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07810a5e0 con 0x7fa0780f67c0 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.954+0000 7fa0757fa700 1 -- 172.21.15.78:0/2067099361 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa064013410 con 0x7fa0780f67c0 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.954+0000 7fa0757fa700 1 -- 172.21.15.78:0/2067099361 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa06401c620 con 0x7fa0780f67c0 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.954+0000 7fa07dd87700 1 -- 172.21.15.78:0/2067099361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07810ab40 con 0x7fa0780f67c0 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.955+0000 7fa07dd87700 1 -- 172.21.15.78:0/2067099361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa078045ad0 con 0x7fa0780f67c0 2023-12-21T03:57:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.955+0000 7fa0757fa700 1 -- 172.21.15.78:0/2067099361 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa06401c8e0 con 0x7fa0780f67c0 2023-12-21T03:57:59.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.956+0000 7fa0757fa700 1 --2- 172.21.15.78:0/2067099361 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa06805c2b0 0x7fa06805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-21T03:57:59.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.956+0000 7fa0757fa700 1 -- 172.21.15.78:0/2067099361 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa064077b40 con 0x7fa0780f67c0 2023-12-21T03:57:59.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.956+0000 7fa07cd85700 1 --2- 172.21.15.78:0/2067099361 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa06805c2b0 0x7fa06805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-21T03:57:59.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.956+0000 7fa07cd85700 1 --2- 172.21.15.78:0/2067099361 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa06805c2b0 0x7fa06805e770 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fa0780f6620 tx=0x7fa06c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-21T03:57:59.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:57:59.960+0000 7fa0757fa700 1 -- 172.21.15.78:0/2067099361 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa06404aa20 con 0x7fa0780f67c0 2023-12-21T03:58:00.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.168+0000 7fa07dd87700 1 -- 172.21.15.78:0/2067099361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa0780fdcb0 con 0x7fa0780f67c0 2023-12-21T03:58:00.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.169+0000 7fa0757fa700 1 -- 172.21.15.78:0/2067099361 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fa06404e040 con 0x7fa0780f67c0 2023-12-21T03:58:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-12-21T03:58:00.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.170+0000 7fa07dd87700 1 -- 172.21.15.78:0/2067099361 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa06805c2b0 msgr2=0x7fa06805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:58:00.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.170+0000 7fa07dd87700 1 --2- 172.21.15.78:0/2067099361 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa06805c2b0 0x7fa06805e770 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fa0780f6620 tx=0x7fa06c009040).stop 2023-12-21T03:58:00.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.170+0000 7fa07dd87700 1 -- 172.21.15.78:0/2067099361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0780f67c0 msgr2=0x7fa0781095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-21T03:58:00.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.170+0000 7fa07dd87700 1 --2- 172.21.15.78:0/2067099361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0780f67c0 0x7fa0781095b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fa064009830 tx=0x7fa064009b30).stop 2023-12-21T03:58:00.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.170+0000 7fa07dd87700 1 -- 172.21.15.78:0/2067099361 shutdown_connections 2023-12-21T03:58:00.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.170+0000 7fa07dd87700 1 --2- 172.21.15.78:0/2067099361 >> [v2:172.21.15.78:6800/2487427889,v1:172.21.15.78:6801/2487427889] conn(0x7fa06805c2b0 0x7fa06805e770 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:58:00.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.170+0000 7fa07dd87700 1 --2- 172.21.15.78:0/2067099361 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0780f5520 0x7fa0781090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:58:00.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.170+0000 7fa07dd87700 1 --2- 172.21.15.78:0/2067099361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0780f67c0 0x7fa0781095b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-21T03:58:00.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.170+0000 7fa07dd87700 1 -- 172.21.15.78:0/2067099361 >> 172.21.15.78:0/2067099361 conn(0x7fa0780f0ca0 msgr2=0x7fa0780603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-21T03:58:00.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.170+0000 7fa07dd87700 1 -- 172.21.15.78:0/2067099361 shutdown_connections 2023-12-21T03:58:00.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-12-21T03:58:00.170+0000 7fa07dd87700 1 -- 172.21.15.78:0/2067099361 wait complete. 2023-12-21T03:58:00.536 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2023-12-21T03:58:00.537 INFO:tasks.cephadm:Teardown begin 2023-12-21T03:58:00.537 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa/tasks/cephadm.py", line 1502, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa/tasks/ceph_manager.py", line 3184, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-12-21T03:58:00.539 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-21T03:58:00.569 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-21T03:58:00.610 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-21T03:58:00.610 DEBUG:teuthology.orchestra.run.smithi078:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-21T03:58:00.628 DEBUG:teuthology.orchestra.run.smithi148:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-21T03:58:00.671 INFO:tasks.cephadm:Stopping all daemons... 2023-12-21T03:58:00.671 INFO:tasks.cephadm.mon.smithi078:Stopping mon.smithi078... 2023-12-21T03:58:00.671 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078 2023-12-21T03:58:00.765 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:58:00 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: audit 2023-12-21T03:58:00.169994+0000 mon.smithi078 (mon.0) 600 : audit [DBG] from='client.? 172.21.15.78:0/2067099361' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:58:00.770 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:58:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: audit 2023-12-21T03:58:00.169994+0000 mon.smithi078 (mon.0) 600 : audit [DBG] from='client.? 172.21.15.78:0/2067099361' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-21T03:58:00.770 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:58:00 smithi078 systemd[1]: Stopping Ceph mon.smithi078 for 5ef0788c-9fb2-11ee-95a5-87774f69a715... 2023-12-21T03:58:00.770 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:58:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:58:00.768+0000 7f62fddca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi078 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-21T03:58:01.012 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078.service' 2023-12-21T03:58:01.038 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:58:00 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078[111706]: debug 2023-12-21T03:58:00.768+0000 7f62fddca700 -1 mon.smithi078@0(leader) e2 *** Got Signal Terminated *** 2023-12-21T03:58:01.038 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:58:00 smithi078 podman[147376]: ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi078 2023-12-21T03:58:01.038 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:58:01 smithi078 systemd[1]: ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi078.service: Succeeded. 2023-12-21T03:58:01.038 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Dec 21 03:58:01 smithi078 systemd[1]: Stopped Ceph mon.smithi078 for 5ef0788c-9fb2-11ee-95a5-87774f69a715. 2023-12-21T03:58:01.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-21T03:58:01.061 INFO:tasks.cephadm.mon.smithi078:Stopped mon.smithi078 2023-12-21T03:58:01.061 INFO:tasks.cephadm.mon.smithi148:Stopping mon.smithi148... 2023-12-21T03:58:01.061 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi148 2023-12-21T03:58:01.398 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:58:01 smithi148 systemd[1]: Stopping Ceph mon.smithi148 for 5ef0788c-9fb2-11ee-95a5-87774f69a715... 2023-12-21T03:58:01.398 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:58:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:58:01.178+0000 7fb096759700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi148 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-21T03:58:01.398 INFO:journalctl@ceph.mon.smithi148.smithi148.stdout:Dec 21 03:58:01 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-mon.smithi148[130747]: debug 2023-12-21T03:58:01.178+0000 7fb096759700 -1 mon.smithi148@1(peon) e2 *** Got Signal Terminated *** 2023-12-21T03:58:01.527 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@mon.smithi148.service' 2023-12-21T03:58:01.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-21T03:58:01.595 INFO:tasks.cephadm.mon.smithi148:Stopped mon.smithi148 2023-12-21T03:58:01.595 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-12-21T03:58:01.595 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.0 2023-12-21T03:58:01.931 INFO:journalctl@ceph.osd.0.smithi078.stdout:Dec 21 03:58:01 smithi078 systemd[1]: Stopping Ceph osd.0 for 5ef0788c-9fb2-11ee-95a5-87774f69a715... 2023-12-21T03:58:01.931 INFO:journalctl@ceph.osd.0.smithi078.stdout:Dec 21 03:58:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.0[120755]: debug 2023-12-21T03:58:01.713+0000 7fefb6a0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-21T03:58:01.931 INFO:journalctl@ceph.osd.0.smithi078.stdout:Dec 21 03:58:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.0[120755]: debug 2023-12-21T03:58:01.713+0000 7fefb6a0d700 -1 osd.0 36 *** Got signal Terminated *** 2023-12-21T03:58:01.931 INFO:journalctl@ceph.osd.0.smithi078.stdout:Dec 21 03:58:01 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.0[120755]: debug 2023-12-21T03:58:01.713+0000 7fefb6a0d700 -1 osd.0 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-21T03:58:02.229 INFO:journalctl@ceph.osd.0.smithi078.stdout:Dec 21 03:58:01 smithi078 podman[147467]: ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.0 2023-12-21T03:58:02.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.0.service' 2023-12-21T03:58:02.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-21T03:58:02.988 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-12-21T03:58:02.988 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-12-21T03:58:02.988 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.1 2023-12-21T03:58:03.283 INFO:journalctl@ceph.osd.1.smithi078.stdout:Dec 21 03:58:03 smithi078 systemd[1]: Stopping Ceph osd.1 for 5ef0788c-9fb2-11ee-95a5-87774f69a715... 2023-12-21T03:58:03.283 INFO:journalctl@ceph.osd.1.smithi078.stdout:Dec 21 03:58:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.1[123909]: debug 2023-12-21T03:58:03.129+0000 7f2418968700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-21T03:58:03.283 INFO:journalctl@ceph.osd.1.smithi078.stdout:Dec 21 03:58:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.1[123909]: debug 2023-12-21T03:58:03.129+0000 7f2418968700 -1 osd.1 36 *** Got signal Terminated *** 2023-12-21T03:58:03.283 INFO:journalctl@ceph.osd.1.smithi078.stdout:Dec 21 03:58:03 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.1[123909]: debug 2023-12-21T03:58:03.129+0000 7f2418968700 -1 osd.1 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-21T03:58:03.566 INFO:journalctl@ceph.osd.1.smithi078.stdout:Dec 21 03:58:03 smithi078 podman[147648]: ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.1 2023-12-21T03:58:04.330 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.1.service' 2023-12-21T03:58:04.369 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-21T03:58:04.369 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-12-21T03:58:04.369 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-12-21T03:58:04.370 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.2 2023-12-21T03:58:04.732 INFO:journalctl@ceph.osd.2.smithi078.stdout:Dec 21 03:58:04 smithi078 systemd[1]: Stopping Ceph osd.2 for 5ef0788c-9fb2-11ee-95a5-87774f69a715... 2023-12-21T03:58:04.732 INFO:journalctl@ceph.osd.2.smithi078.stdout:Dec 21 03:58:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.2[127059]: debug 2023-12-21T03:58:04.561+0000 7fb06189a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-21T03:58:04.732 INFO:journalctl@ceph.osd.2.smithi078.stdout:Dec 21 03:58:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.2[127059]: debug 2023-12-21T03:58:04.562+0000 7fb06189a700 -1 osd.2 36 *** Got signal Terminated *** 2023-12-21T03:58:04.732 INFO:journalctl@ceph.osd.2.smithi078.stdout:Dec 21 03:58:04 smithi078 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.2[127059]: debug 2023-12-21T03:58:04.562+0000 7fb06189a700 -1 osd.2 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-21T03:58:05.066 INFO:journalctl@ceph.osd.2.smithi078.stdout:Dec 21 03:58:04 smithi078 podman[147827]: ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.2 2023-12-21T03:58:05.824 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.2.service' 2023-12-21T03:58:05.864 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-21T03:58:05.865 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-12-21T03:58:05.865 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-12-21T03:58:05.865 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.3 2023-12-21T03:58:06.156 INFO:journalctl@ceph.osd.3.smithi148.stdout:Dec 21 03:58:05 smithi148 systemd[1]: Stopping Ceph osd.3 for 5ef0788c-9fb2-11ee-95a5-87774f69a715... 2023-12-21T03:58:06.157 INFO:journalctl@ceph.osd.3.smithi148.stdout:Dec 21 03:58:06 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.3[133928]: debug 2023-12-21T03:58:06.016+0000 7f22b14a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-21T03:58:06.157 INFO:journalctl@ceph.osd.3.smithi148.stdout:Dec 21 03:58:06 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.3[133928]: debug 2023-12-21T03:58:06.016+0000 7f22b14a9700 -1 osd.3 36 *** Got signal Terminated *** 2023-12-21T03:58:06.157 INFO:journalctl@ceph.osd.3.smithi148.stdout:Dec 21 03:58:06 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.3[133928]: debug 2023-12-21T03:58:06.016+0000 7f22b14a9700 -1 osd.3 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-21T03:58:06.412 INFO:journalctl@ceph.osd.3.smithi148.stdout:Dec 21 03:58:06 smithi148 podman[142643]: ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.3 2023-12-21T03:58:07.071 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.3.service' 2023-12-21T03:58:07.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-21T03:58:07.111 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-12-21T03:58:07.111 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-12-21T03:58:07.112 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.4 2023-12-21T03:58:07.413 INFO:journalctl@ceph.osd.4.smithi148.stdout:Dec 21 03:58:07 smithi148 systemd[1]: Stopping Ceph osd.4 for 5ef0788c-9fb2-11ee-95a5-87774f69a715... 2023-12-21T03:58:07.413 INFO:journalctl@ceph.osd.4.smithi148.stdout:Dec 21 03:58:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.4[137176]: debug 2023-12-21T03:58:07.280+0000 7f0669a65700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-21T03:58:07.413 INFO:journalctl@ceph.osd.4.smithi148.stdout:Dec 21 03:58:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.4[137176]: debug 2023-12-21T03:58:07.280+0000 7f0669a65700 -1 osd.4 36 *** Got signal Terminated *** 2023-12-21T03:58:07.414 INFO:journalctl@ceph.osd.4.smithi148.stdout:Dec 21 03:58:07 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.4[137176]: debug 2023-12-21T03:58:07.280+0000 7f0669a65700 -1 osd.4 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-21T03:58:07.765 INFO:journalctl@ceph.osd.4.smithi148.stdout:Dec 21 03:58:07 smithi148 podman[142825]: ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.4 2023-12-21T03:58:08.496 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.4.service' 2023-12-21T03:58:08.537 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-21T03:58:08.537 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-12-21T03:58:08.537 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-12-21T03:58:08.537 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.5 2023-12-21T03:58:08.765 INFO:journalctl@ceph.osd.5.smithi148.stdout:Dec 21 03:58:08 smithi148 systemd[1]: Stopping Ceph osd.5 for 5ef0788c-9fb2-11ee-95a5-87774f69a715... 2023-12-21T03:58:08.765 INFO:journalctl@ceph.osd.5.smithi148.stdout:Dec 21 03:58:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.5[140324]: debug 2023-12-21T03:58:08.666+0000 7fa6117fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-21T03:58:08.765 INFO:journalctl@ceph.osd.5.smithi148.stdout:Dec 21 03:58:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.5[140324]: debug 2023-12-21T03:58:08.666+0000 7fa6117fb700 -1 osd.5 36 *** Got signal Terminated *** 2023-12-21T03:58:08.765 INFO:journalctl@ceph.osd.5.smithi148.stdout:Dec 21 03:58:08 smithi148 ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.5[140324]: debug 2023-12-21T03:58:08.666+0000 7fa6117fb700 -1 osd.5 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-21T03:58:09.236 INFO:journalctl@ceph.osd.5.smithi148.stdout:Dec 21 03:58:08 smithi148 podman[143006]: ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715-osd.5 2023-12-21T03:58:09.913 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ef0788c-9fb2-11ee-95a5-87774f69a715@osd.5.service' 2023-12-21T03:58:09.952 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-21T03:58:09.952 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-12-21T03:58:09.952 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 --force --keep-logs 2023-12-21T03:58:13.055 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 --force --keep-logs 2023-12-21T03:58:39.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-21T03:58:39.319 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-21T03:58:39.351 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-21T03:58:39.353 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050/remote/ubuntu@smithi078.front.sepia.ceph.com/crash 2023-12-21T03:58:39.356 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/crash -- . 2023-12-21T03:58:39.411 INFO:teuthology.orchestra.run.smithi078.stderr:tar: /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/crash: Cannot open: No such file or directory 2023-12-21T03:58:39.411 INFO:teuthology.orchestra.run.smithi078.stderr:tar: Error is not recoverable: exiting now 2023-12-21T03:58:39.413 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050/remote/ubuntu@smithi148.front.sepia.ceph.com/crash 2023-12-21T03:58:39.414 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/crash -- . 2023-12-21T03:58:39.473 INFO:teuthology.orchestra.run.smithi148.stderr:tar: /var/lib/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/crash: Cannot open: No such file or directory 2023-12-21T03:58:39.473 INFO:teuthology.orchestra.run.smithi148.stderr:tar: Error is not recoverable: exiting now 2023-12-21T03:58:39.474 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-21T03:58:39.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-21T03:58:39.509 INFO:teuthology.orchestra.run.smithi078.stderr:grep: /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph.log: No such file or directory 2023-12-21T03:58:39.512 INFO:tasks.cephadm:Compressing logs... 2023-12-21T03:58:39.512 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-21T03:58:39.554 DEBUG:teuthology.orchestra.run.smithi148:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-21T03:58:39.583 INFO:teuthology.orchestra.run.smithi148.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-21T03:58:39.583 INFO:teuthology.orchestra.run.smithi148.stderr:: No such file or directory 2023-12-21T03:58:39.583 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-volume.log 2023-12-21T03:58:39.583 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mgr.smithi148.tgpyxd.log 2023-12-21T03:58:39.583 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-21T03:58:39.583 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.3.log 2023-12-21T03:58:39.584 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-volume.log: /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mgr.smithi148.tgpyxd.log: gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mon.smithi148.log 2023-12-21T03:58:39.584 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.4.log 2023-12-21T03:58:39.584 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mon.smithi148.log: gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.5.log 2023-12-21T03:58:39.584 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.4.log: /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.5.log: 89.6% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-volume.log.gz 2023-12-21T03:58:39.588 INFO:teuthology.orchestra.run.smithi148.stderr: 92.7% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mgr.smithi148.tgpyxd.log.gz 2023-12-21T03:58:39.589 INFO:teuthology.orchestra.run.smithi078.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-12-21T03:58:39.590 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-21T03:58:39.590 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-volume.log 2023-12-21T03:58:39.590 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.2.log 2023-12-21T03:58:39.590 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.0.log 2023-12-21T03:58:39.590 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mon.smithi078.log 2023-12-21T03:58:39.591 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mgr.smithi078.zidbgq.log 2023-12-21T03:58:39.591 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mon.smithi078.log: gzip -5 --verbose -- /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.1.log 2023-12-21T03:58:39.591 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mgr.smithi078.zidbgq.log: /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.1.log: 90.1% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-volume.log.gz 2023-12-21T03:58:39.601 INFO:teuthology.orchestra.run.smithi078.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-21T03:58:39.625 INFO:teuthology.orchestra.run.smithi148.stderr: 93.2% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.5.log.gz 2023-12-21T03:58:39.625 INFO:teuthology.orchestra.run.smithi148.stderr: 93.2% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.4.log.gz 2023-12-21T03:58:39.639 INFO:teuthology.orchestra.run.smithi148.stderr: 92.5% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.3.log.gz 2023-12-21T03:58:39.645 INFO:teuthology.orchestra.run.smithi078.stderr: 92.6% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.1.log.gz 2023-12-21T03:58:39.661 INFO:teuthology.orchestra.run.smithi078.stderr: 93.0% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.2.log.gz 2023-12-21T03:58:39.667 INFO:teuthology.orchestra.run.smithi078.stderr: 92.6% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-osd.0.log.gz 2023-12-21T03:58:39.693 INFO:teuthology.orchestra.run.smithi148.stderr: 93.0% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mon.smithi148.log.gz 2023-12-21T03:58:39.695 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-12-21T03:58:39.696 INFO:teuthology.orchestra.run.smithi148.stderr:real 0m0.124s 2023-12-21T03:58:39.696 INFO:teuthology.orchestra.run.smithi148.stderr:user 0m0.252s 2023-12-21T03:58:39.696 INFO:teuthology.orchestra.run.smithi148.stderr:sys 0m0.029s 2023-12-21T03:58:39.758 INFO:teuthology.orchestra.run.smithi078.stderr: 93.1% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mgr.smithi078.zidbgq.log.gz 2023-12-21T03:58:39.928 INFO:teuthology.orchestra.run.smithi078.stderr: 90.5% -- replaced with /var/log/ceph/5ef0788c-9fb2-11ee-95a5-87774f69a715/ceph-mon.smithi078.log.gz 2023-12-21T03:58:39.929 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-12-21T03:58:39.930 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m0.356s 2023-12-21T03:58:39.930 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m0.720s 2023-12-21T03:58:39.930 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m0.046s 2023-12-21T03:58:39.930 INFO:tasks.cephadm:Archiving logs... 2023-12-21T03:58:39.930 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050/remote/ubuntu@smithi078.front.sepia.ceph.com/log 2023-12-21T03:58:39.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-21T03:58:40.150 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050/remote/ubuntu@smithi148.front.sepia.ceph.com/log 2023-12-21T03:58:40.150 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-21T03:58:40.233 INFO:tasks.cephadm:Removing cluster... 2023-12-21T03:58:40.233 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 --force 2023-12-21T03:58:40.524 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5ef0788c-9fb2-11ee-95a5-87774f69a715 --force 2023-12-21T03:58:40.821 INFO:tasks.cephadm:Removing cephadm ... 2023-12-21T03:58:40.822 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-21T03:58:40.840 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-21T03:58:40.858 INFO:tasks.cephadm:Teardown complete 2023-12-21T03:58:40.859 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa/tasks/cephadm.py", line 1502, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa/tasks/ceph_manager.py", line 3184, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-12-21T03:58:40.943 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=942851d5bc0044ae843dc95acc07eac0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa/tasks/cephadm.py", line 1502, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_639f0137318def3e5f393094e5494880043a8f32/qa/tasks/ceph_manager.py", line 3184, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-12-21T03:58:40.945 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-21T03:58:40.956 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-21T03:58:40.988 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-21T03:58:40.988 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-21T03:58:41.002 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-21T03:58:41.082 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-21T03:58:41.092 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-21T03:58:41.092 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-21T03:58:41.095 DEBUG:teuthology.orchestra.run.smithi148:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-21T03:58:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2023-12-21T03:58:41.111 INFO:teuthology.orchestra.run.smithi148.stderr:bash: ntpq: command not found 2023-12-21T03:58:41.115 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-21T03:58:41.116 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2023-12-21T03:58:41.116 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 +5868ns[-1478ns] +/- 96ms 2023-12-21T03:58:41.116 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -473us[ -480us] +/- 61ms 2023-12-21T03:58:41.116 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +489us[ +489us] +/- 68ms 2023-12-21T03:58:41.116 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-21T03:58:41.119 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-21T03:58:41.119 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2023-12-21T03:58:41.119 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +8189ns[+3272ns] +/- 96ms 2023-12-21T03:58:41.119 INFO:teuthology.orchestra.run.smithi148.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -463us[ -468us] +/- 61ms 2023-12-21T03:58:41.119 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +528us[ +523us] +/- 68ms 2023-12-21T03:58:41.119 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-21T03:58:41.120 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-21T03:58:41.129 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-21T03:58:41.129 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-21T03:58:41.146 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-21T03:58:41.250 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-21T03:58:41.338 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-21T03:58:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-21T03:58:41.370 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-21T03:58:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-21T03:58:41.402 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-21T03:58:41.412 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-21T03:58:41.449 INFO:teuthology.task.internal:Duration was 1900.786415 seconds 2023-12-21T03:58:41.449 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-21T03:58:41.482 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-21T03:58:41.483 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-21T03:58:41.485 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-21T03:58:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-21T03:58:41.531 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-21T03:58:41.828 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-21T03:58:41.828 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2023-12-21T03:58:41.829 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-21T03:58:41.851 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2023-12-21T03:58:41.851 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-21T03:58:41.875 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-21T03:58:41.875 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-21T03:58:41.894 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-21T03:58:41.964 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-21T03:58:41.964 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-21T03:58:41.967 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-21T03:58:42.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-21T03:58:42.076 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-21T03:58:42.076 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-21T03:58:42.104 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-21T03:58:42.132 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-21T03:58:42.142 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-21T03:58:42.147 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-21T03:58:42.170 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2023-12-21T03:58:42.199 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2023-12-21T03:58:42.225 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-21T03:58:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-21T03:58:42.259 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-21T03:58:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-21T03:58:42.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-21T03:58:42.295 INFO:teuthology.task.internal:Transferring archived files... 2023-12-21T03:58:42.296 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050/remote/smithi078 2023-12-21T03:58:42.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-21T03:58:42.341 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050/remote/smithi148 2023-12-21T03:58:42.342 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-21T03:58:42.384 INFO:teuthology.task.internal:Removing archive directory... 2023-12-21T03:58:42.384 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-21T03:58:42.386 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-21T03:58:42.434 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-21T03:58:42.447 INFO:teuthology.task.internal:Not uploading archives. 2023-12-21T03:58:42.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-21T03:58:42.464 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-21T03:58:42.465 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-21T03:58:42.468 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-21T03:58:42.483 INFO:teuthology.orchestra.run.smithi078.stdout: 525838 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 21 03:58 /home/ubuntu/cephtest 2023-12-21T03:58:42.492 INFO:teuthology.orchestra.run.smithi148.stdout: 525838 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 21 03:58 /home/ubuntu/cephtest 2023-12-21T03:58:42.493 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-21T03:58:42.507 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-21T03:58:42.566 INFO:teuthology.nuke:Checking targets against current locks 2023-12-21T03:58:42.600 DEBUG:teuthology.nuke:shortname: smithi078 2023-12-21T03:58:42.601 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-21T03:58:42.624 DEBUG:teuthology.nuke:shortname: smithi148 2023-12-21T03:58:42.624 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-21T03:58:42.630 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-21 03:18:26.736310', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLB67fRCx7IGXvwXku5ZxZXZhxC4PAwTiz0aLgXJ+n6xIJKA/R+CC+J9OQzwkIFbkdiLq5yNDGyUz2FEmUbr2EI='} 2023-12-21T03:58:42.654 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-20_19:57:16-rados-wip-yuri7-testing-2023-12-20-0808-pacific-distro-default-smithi/7499050', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-21 03:18:26.737734', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJDqnnwX0RmahZB89gv7kXzLnOfDTZrhPFy++7LcxJyHtGk7gepq0QT8PDnO8L5uJ41QKnES5EEOvU0oDScyl90='} 2023-12-21T03:58:42.682 INFO:teuthology.orchestra.console.smithi078:Power off 2023-12-21T03:58:42.683 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-21T03:58:42.710 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2023-12-21T03:58:42.711 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-21T03:58:42.732 INFO:teuthology.orchestra.console.smithi148:Power off 2023-12-21T03:58:42.733 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-21T03:58:42.743 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-12-21T03:58:42.756 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2023-12-21T03:58:42.757 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-21T03:58:42.781 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-12-21T03:58:46.745 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-21T03:58:46.782 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-21T03:58:46.856 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-12-21T03:58:46.894 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-12-21T03:58:50.861 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-21T03:58:50.895 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-21T03:58:50.972 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2023-12-21T03:58:51.007 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-12-21T03:58:51.073 INFO:teuthology.orchestra.console.smithi078:Power off completed 2023-12-21T03:58:55.013 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-21T03:58:55.123 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2023-12-21T03:58:55.224 INFO:teuthology.orchestra.console.smithi148:Power off completed 2023-12-21T03:58:55.328 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1900.7864151000977 failure_reason: timeout expired in wait_until_healthy flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=942851d5bc0044ae843dc95acc07eac0 status: fail success: false 2023-12-21T03:58:55.329 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-21T03:58:55.415 INFO:teuthology.run:FAIL